I only retract the swing keel when hauling the boat, or in shallow water, ie: near a beach or sneaking up a shallow creek.
A few years ago on the west coast a swing keel sailboat with a large number of people on board, capsized in a harbor. Don't recall the exact details but the swing keel was reported to be retracted.
Jackdaw is correct on the design factor.