From what I can tell from my LI Maptech chart book (help T. Weitz!)
There is an anchorage on the north side of Robbins Island which would be good in W, N, SW, or S winds but would be open to E, NE or N winds. It is a little closer to Cutchogue on the north fork but it is the closest 'official' anchorage to North Sea. There are anchorages shown on either side of North Haven Peninsula, one which is close to Noyack (pretty close to North Sea) and the other on the Sag Harbor (or eastern) side of the peninsula.
I don't know what your Pearson 32 draws but I have driven along Noyac Road many times past a marina at North Sea Harbor that seems to mostly have motor boats in it. My chart book does not have an inset for North Sea Harbor so I would call ahead for local knowledge. It looks shallow on the chart I do have which is more 'macro' in detail.
Similarly another 'bay' on the Great Peconic side of Robbins Island called Bullhead Bay looks unpromising from my charts. Wooley Pond looks the most promising of the inlets from what I can tell. Otherwise you are anchoring out in Little Peconic Bay off North Sea and pretty open to the prevailing SW winds.
Perhaps Ted Weitz who keeps his boat in 3 Mi. Harbor and is on this forum can give you more first hand advice.
Download chart 12358 from the following link and you will see for yourself: http://ocsdata.ncd.noaa.gov/BookletChart/AtlanticCoastBookletCharts.htm
My guess is that they call the southern channel below Robbins Island 'South Race' is because there is some current going through there on a regular basis.
Sag Harbor is not all that far from North Sea either.
Good luck and enjoy. It is pretty back there.