Here's my guess: You have a blocked holding tank vent that's causing the tank--the whole system--to become so seriously pressurized by that it's forcing effluent to escape wherever it can. That may be a y-valve, the tank discharge fitting, inlet fitting, vent fitting... I'm surprised it hasn't erupted in the toilet! The only reason it hasn't is because it's found an easier way out. If I'm right, the only reason you haven't had the fun of getting soaked by a geyser out the deck pumpout fitting when you loosen the cap is because you don't try to pumpout before leaving the boat, you wait till you return. By then the pressure has been relieved enough to let the pumpout get at least some of what's left in the tank before it pulls a vaccuum that won't allow it to remove any more.
A manual toilet that becomes increasingly hard to pump or an electric toilet that starts to "burp" and/or "spit up" after you've used it a bit would confirm my guess.
So do a couple of things: Immediately upon returning to the boat, BEFORE you use the head or try to pump out, check your tank vent...start with the vent thru-hull. Use a screwdriver blade, ice pick or whatever works to clean it out. If there's a screen in it, knock it out...screens cause more problems than they solve. If the tank is less than half full, remove the vent line from it and clean out that end of the vent line and the vent fitting on the tank. Might be a good idea to open the deck pumpout cap before you remove any hoses, just in case there's still any pressure left in the system.
NOW you should be able to pump out and THOROUGHLY rinse out the tank. Do that before you leave the boat this time. Then fill the tank with clean water to which you've added red food coloring. When you return the boat, you should see a pink stain where any leaks are, but now that the system is no longer being pressurized, your bilges should be dry.
All this is just a guess, but my experience and limited psychic powers tell me I'm at least on the right track, if I haven't nailed it yet.