Decades ago (ok, make that many decades ago!) I used to sail over there in a Rhodes 19. That was a boat with a keel. Looking back on it, I think I was foolish to do that, because the boat was not self-bailing. However, capsizing it was not a major consideration, and I did live to tell the tale.
I assume your boat has positive flotation, but capsizing remains a possibility. If you have a good weather window, you will be OK. If you don't--well, you are taking a risk, possibly a big one.
I've made that crossing hundreds of times in the nearly 55 years I have been sailing there (I'm a San Pedro native, sailing since age 12), and I can tell you that conditions can get very ugly in that channel. You can also have a very benign crossing, and everything in between. The question is, "do you feel lucky?"
I wouldn't hesitate to do the trip in a Cal 20, and in fact did so many, many times in high school on a friend's boat, as well as later when I owned one myself. But even a Cal 20 is a much more seaworthy craft that yours. I don't think I'd do it in a 19' centerboard boat, as you are proposing.