I've read this here a number of times from different folks.
WTF?
MILES out of your way?
Nonsense.
All you need to do is duck the racers, which takes maybe a 5 degree course change. You do NOT have to stay off the entire race course. You just have to avoid the boats that are ON the race course.
Where some people get these crazy ideas is beyond me.
I've sailed on SF Bay for over 35 years. There are usually tons of races every single Saturday & Sunday, ALL YEAR ROUND. You simply cannot avoid the races, because they are all over the Bay. You don't even have to know the courses or the marks. Just be polite and duck them, do NOT head up, like Jackdaw said, it's confusing. And you can only duck one boat at a time, deal with the second one after you pass the first one.
Close to some people means a mile. That's simply nuts.