Thinking about this thread, it occurred to me that the pleasure one gets from sailing is dependent upon a lot of factors. One factor is matching the boat to the use that is most pleasurable.
I've owned 3 boats (we don't talk about the flying scot), a Tanzer 22, a Sabre 30, and a Sabre 362. I loved and hated each of them. The Tanzer was a great boat for day sailing and sailing in lighter conditions. She was (and still is) an easy boat to rig and sail and maintenance costs were low as were upgrade costs. But, she was not the most comfortable boat to cruise on. On one trip at a much younger age, after banging my head on the cabin top yet again, the refrain on the boat was "We need a bigger boat!"
My current boat, the Sabre 362, is a great dock condo, a delightful boat to cruise on, and handles 20 Kt breezes with ease. She is a very comfortable boat to cruise for a week or more. But, hoisting a 60# mainsail up a 44 foot hoist is a bear. As is docking and maneuvering in close quarters. Rarely do we hop on her and go sailing for an hour or two. If we go, it is for a much longer time. And we plan to take a year long trip down the ICW to the Bahamas in the near future. This will be a great boat.
The point is to match your interests in sailing with the boat. A smaller or faster boat may suit your interests more than the boat you have. There are some small boats that perform well in lighter air, but are a bear in heavier air. Find the sweet spot for your area and your interests. If we were not planning a longer trip, the Sabre 30 would have been the boat for us.
As always, YMMV.