I love the comfort and "weekend range" of my Bavaria 38E such that it's tough to think about down-sizing. I put on a smaller headsail (roller furling/reefing) for easier tacking/trimming, and installed at 4-point lazy-jack system that allows me to drop the mainsail quickly w/o making a big mess of sail on the coach top and in the cockpit. It'll ride there until I can strap on a sail tie. Add autopilot and electric windlass and the only thing left that seems demanding is hoisting the main and moving the main traveler to further to windward under load. Before I'd downsize, I'd install an electric winch in a strategic location to assist w/hoisting and some sheeting. The boat for the duffers that does it all is the IP 41-ft SP cruiser; but, you'll need a lot of green.
http://www.yachtworld.com/boats/2010/Island-Packet-SP-Cruiser-3115983/Annapolis/MD/United-States?refSource=browse listing#.WpY2ydQrJpg
http://www.yachtworld.com/boats/2010/Island-Packet-SP-Cruiser-3115983/Annapolis/MD/United-States?refSource=browse listing#.WpY2ydQrJpg