I owned a Seafarer 29 with a hinged mast step. I assume yours is not hinged? My best guess is that there is a deck plate mounted to the cabin top that fits inside the mast. Since there is movement this suggests that the deck plate may be loose. Also, the mast should fit flat to the deck on all sides.
There's a couple possibilities here. First, I suspect the mast may be raked forward too much allowing the aft side to not be seated. You could let out on the forestay and tighten the backstay to rake the mast back until the base of the mast is seated better.
The other possibility is that there is a lack of support under the mast's base plate. Check to make sure that the compression post is in the proper position.
Dropping the mast would be advised for many reasons. It will allow you to inspect the deck plate as well as all the fittings and rigging on the mast. It will also give you a good chance to check the wiring for your steaming and anchor lights and switch to LED if interested.
We have a very active Seafarer owners' group on Facebook that is monitored better than this forum. Our resident S24 owner/authority is Rusty Lillico from Hawaii. I suggest that you also join this forum. Here's a link. https://www.facebook.com/groups/seafareryachts/
If you're not on Facebook let me know and I'll message Rusty and ask him to respond here.