I agree with Maine Sail.
As the business consultants say in answer to any question: "It depends."
Last Spring we got beat up pretty badly taking 7 hours to get from Nantucket to Martha's Vineyard (28nm) "riding" a 2kt ebb against 30kts from the W. They were "only 4ft seas." (And about 4-5 sec apart in shallow Nantucket Sound - less to us, due to our going against them.)
My wife's dirty look told me that that was certainly too much, when Rick at Black Dog Wharf said: "Aren't you the hearty sailors! I tried to take my board out, but it was too rough for me."
This past Sunday, coming back from our annual September Maine cruise, we surfed 6ft almost-following seas in NE 18-22 from Portland to Kennebunkport in 4hrs (30nm). Those "6ft seas" were nothing - although I had to keep my wife on pot-buoy watch to prevent her from looking astern to see them coming - it was a blast.
Wave height is less important than apparent period (actual period modified by boat velocity with or against). So 10 fathom coastal waters with gradual, sandy beaches (S of Cape Elizabeth) are different than 20 fathoms-plus with bold shores (coasts N of Cape Elizabeth).
Simply having wind also is important; sailing 8ft swells on-the-beam are a piece of cake when there's enough wind to keep the boat heeled. Motoring in them in light air - even with mainsail up to provide a stabilizer fin - can get unpleasant for less experienced stomachs.
A lot also depends on how you're rigged.
When I took delivery of Persephone (8250nm ago, in 2005) I had Doyle make me a suite of sails good from 3-to-30kts, and I can personally attest to the fact that they meet the spec. (490 sq-ft all-full-batten mainsail with 3 reef points and 305 sq-ft jib with foam luff - both of 8.8oz Type 52 Dacron - plus 850sq-ft asymmetric spinnaker of 3/4 oz Nylon with snuffer sock.)
I have every control imaginable led aft to the cockpit, and only go forward to rig the spinnaker or whisker pole. I also have permanent SS jacklines rigged for use with my 4ft tethers that keep me near the centerline and away from the lifelines (which are also laced with wire forward of the shrouds, for unplanned foredeck adventures).
(I added the permanent centerline jackline after my boom parted from the gooseneck on a 2-reefed beam reach after rounding Cape Small in to Bailey's Island a few years ago. I didn't find my "conventional wisdom" sidedeck jacklines confidence-inspiring while on the cabin-top trying to jam a screwdriver into the holes vacated by the errant hinge-pin.)
Having confidence in your boat's reactions to stressful conditions - through personal experience, rather than hearsay - is extremely important.
I have a boat in which I've been pinned on one side by a summer thunderstorm's downdraft, and so know how she handles such things (calmly fore-reaching with a dry cockpit, until I could crawl forward to release the mainsheet).
As a result, until it's gale force (which requires me to avoid wind-on-the beam to have good steering - even with my big spade rudder), I generally don't pay a lot of attention to wind speed per se, but more to sea state in deciding whether or not to go out. (The photo in my avatar was taken out by the Isles of Shoals in Force 6 gusting to 7, but only 2-3ft seas.)
I always progressively reef-down the main, rather than simply dropping. Then, when I go out, I can hoist it to 3-reefed to start in heavy air, before putting up more sail by shaking out reefs. Much better than full-hoist followed by frantic reefing exercises. (I also have a mainsail downhaul for dousing quickly in heavy air.)
Before going out, I look at relevant dial-a-buoys for wave height and period (real data is better than forecasts), and at whether or not I'm single-handing or must account for the comfort of others.
Whatever your current threshold, I firmly believe in progressively increasing the conditions in which you practice (single-handed, to avoid complications) until you are comfortable with your boat and your ability to handle her in conditions beyond those in which you plan to normally sail.
Mother Atlantic is fickle, and doesn't check in with the National Weather Service when she changes her mind. (Sunday's forecast was NE14 with 3-4ft seas.)
Fair winds,
Al - s/v Persephone