Lots of good info above. My 2 cents: A good broker will work with you on getting a boat that suits your needs and not sell you something just to make a sale and put money in his/her pocket. And, a good broker might not have the boat of your choice, so you may not end up doing business with that person, or he/she can search and find your boat. Lastly, a good broker should not get bored with frivolous questions that might waste his/her time. A good broker will compare differences in boats on all levels and help you weave through the multitude of questions you may have. If you don't do business with that broker this time around, he may enjoy your business the next time because you had a great experience. Regarding the boat: As asked above, you need to set some priorities as to how you wish to use the boat, where you are going to sail, etc. Coastal cruising, passage-making and where you specifically will visit. For example, in the Bahamas you don't want a very deep draft. 5.5' is ideal. You can get away with 6.5' but that might limit where you can go and you might have to wait for high tide. Same with SW Florida. If you are going to do long distance cruising - offshore - then you need a boat built for bad conditions. There are a number of comfortable, stylish vessels out there, but if they are not classified for offshore sailing, you don't want them. Look for Class A Ocean boat designed for extended voyages where conditions may exceed wind force 8 (Beaufort scale) and significant wave heights, abnormal conditions and boats that are basically self-sufficient. Class B Offshore is very similar. Class C is for inshore voyages in coastal waters to include wind force 6 and significant wave heights. Class D is for sheltered and inland waters, bays, estuaries, lakes, rivers - winds to force 4......
So, boats designed for ocean or offshore purposes will be built stronger with better hull to deck joints, windows that can take heavy seas, cockpits that have sufficient drains (scuppers) and a bridge deck sufficient to not let water down through the companionway, companionway drop boards built to withstand the worst of conditions, rigs and standing rigging that are sufficient for extreme conditions, winches that are adequate in size and enough for all the tasks required, an inboard engine that has enough horse power to get you through extreme conditions, sea friendly cockpits for offshore use rather than just cocktailing, lifelines, stanchions and railings designed to withstand heavier abuse, electronics to accommodate all your navigation, communications and call for help needs, etc. etc.
Comfort: Obviously enough berths to accommodate everyone. Tanks large enough to supply you with enough potable water and fuel to take you long distances when the wind isn't blowing. Storage so you can put things away and not have gear taking up living areas of your boat. Hand rails and other things to keep you safe below when in heavy situations.....I am simply just briefly going over things to think about.
Obviously for weekend sailing, and not so ambitious outings, you don't need a boat built for the North Sea if you aren't going to sail in areas where you will be exposed to extreme conditions.
You mentioned center cockpit boats.....I like them because usually there is an awesomely huge aft cabin and head and shower area, and behind the cockpit itself there is a large deck from where you can fish. When you catch something, you can skin and clean them there and not get any of the fish guts, etc. in the cockpit which is a problem with an aft cockpit boat.
Someone mentioned multihulls. They are super comfortable with lots of living space and may accommodate what your needs are, but maybe not sail around the world in one. Also are harder to find dockage for because of their beam. But, everytime you go into a dock it costs you money. When you anchor out, it is free for the most part, which brings up another topic: Ground tackle. You need the proper anchors, rode and chain (and maybe a lot of chain), electric windlass, etc. so you know you will stay put when you drop your anchor(s).
Also to consider is your cruising budget and taking care of the boat. The bigger the boat, the higher the price of most everything, like sails dockage, haul out, storage, bottom painting, etc. When in a marina, the boat seems too big but when out to sea, it's never big enough. You want to get something easy enough to handle based on how many people will be on board. Sail handling equipment like roller furling headsails and mainsails make a lot of sense on larger vessels where sails can be very heavy. Auto pilots let you leave the cockpit to tend to things temporarily on deck.
Of course you need the required safety gear and even maybe a life raft. This all can add up.
A good generator for electricity off shore is a good thing too.
And adequate refrigeration/freezer capabilities for longer voyages is a must.
Hope this helps and gives you things to think about. Good luck.