Open up the top of the pedestal, remove the compass and section that it sits on, then check out all the moving parts. Had a similar problem with our boat when we took possession in 2002. The broker took us (owner and prospect) out for a sea trial. He had trouble shifting and banged up his hand doing so. Probably a minor adjustment. Bought the boat, and before adding mid ship cleats, went our for a few sails. Also had trouble shifting, but then coming back from one of our trips as I approached our slip slowly, placed it into neutral, then just before touching a burst of reverse throttle to stop the boat, a clunk just as I shifted back into neutral and whoa, no shifter.
After removing the top parts of the pedestal, upon close inspection, the clevis that threads onto the end of the shifter cable broke from metal fatigue. The reason it failed was because it had either not been placed into its guide originally, or had popped out at some point causing the shifter lever to burden the clevis at an uneven angle. That guide secures the end of the cable sheath to prevent the clevis from wobbling about when changing gears. After replacing the clevis and properly installing it in the guide, gear changing has become, well, normal. That was in 2002 and shifting has been normal ever since.
Please let me know what you find during your inspection.