"Let's address the overboard side first: If your toilet is below waterline, sea water is rising in the bowl to the waterline. The solution: a vented loop in the toilet discharge line that's at least 6-8" above waterline
at max heel, not just when the boat is at rest...which puts about 2-3 FEET above the bowl on most sailboats. Put it immediately after the toilet...iow, aim the discharge fitting straight up. That requires a 90 degree discharge fitting, if yours is a straight fitting call Jabsco to order a 90. 978-282-5246 is the direct line to Paul Campagna, Jabsco's tech support guy.
You also need a vented loop in the flush water intake line...it does NOT go between the thru-hull and pump...it MUST be installed between the pump and the bowl, which will require replacin the short piece of hose that connects them with two hoses long enough to put it the same height above the toilet as the discharge loop. You'll see illustrations of the correct locations for vented loops in the installation instructions:
Jabsco Manual Twist & Lock owners manual Also see the photos I've attached.
Now let's talk about the backup from the holding tank: I'm betting that it's an uphill run from the toilet to the tank and that you're not flushing long enough in the dry mode for the flushes to get there, so they're running back to the bowl. Tank contents may also be spilling into the toilet discharge line when the boat is heeled. The vented loop in the toilet discharge line will keep water from getting TO the bowl from the tank and the thru-hull, but a li'l doodad called a joker valve (one way valve in the toilet discharge fitting...see the exploded drawing in the instruction manual) is needed to keep it OUT of the bowl, and yours is obviously worn out.
Joker valves should be replaced in all toilets at least annually.
The vented loop in the toilet discharge line can actually increase the number of flushes your tank can hold while reducing the number of times you have to pump the toilet once you learn how to use the dry mode to do more than just push the last bit of water out of the bowl: Any toilet that's working anywhere close to spec--which includes having a joker valve in good shape--can move bowl contents VERTICALLY up to 4' in the dry mode....more than enough to move 'em over the top of the loop...then gravity can get it the rest of the way to the tank. If your toilet can't push the bowl contents over the loop, replace the joker valve. So--ahead of use, pump a couple of times to wet the bowl if for only urine and TP ...add a couple of beer cupfuls of water from the sink ahead of solid waste. Pump enough times in the DRY mode to push the bowl contents over the top of the loop, then switch to wet for only a few flushes to rinse the line behind the flush.
One more thing: although my book is titled "Get Rid of Boat Odors" (see link in my signature), it's actually a comprehensive "marine toilets and holding tanks 101" manual that will teach you how to prevent 99% of problems instead of having to fix 'em. It's available from the sbo.online store....just click on the link. And I'm always glad to answer any questions it doesn't."