Check the Float Switch
The float switch may be gunked up with slime coat. Sometimes this slime will prevent the bilge pump from coming on or being shut off. Look at all wiring connections also for anything that looks abnormal.
If you don't make a habit of keeping the bilge empty of water and dry, then you should at the least make a practice of flushing the bilge and cleaning it out with an approved bilge soap/cleaner and doing a good rinse with fresh water. Most of us keep ours dry.
You should be able to reach down to the switch and notice a couple round knobs sticking out on each side of the float switch if it isn't an automatic switch. Just grab ahold of one of those and rock/roatate it back and forth to see if it shuts the pump off. If if does, you found the cause of your problem.
My suspect would be the float switch or bad wiring.
Good luck, at least it runs, the undesired alternative of course would be it not working and your worry of the boat sinking. See how much better off you are

?