Unless the tank is full to overflowing, there's no way that waste IN the tank can back up into the toilet 'cuz the inlet fitting is on the top of the tank and tank contents can't jump up into it. The ONLY exception would be waste spilling into the toilet discharge hose when the boat is heeled...but that can only happen when the tank is at least half full and only while the boat is heeled over pretty hard. So what's running back can only be waste in the toilet discharge line that back pressure is preventing from making into the toilet. And, as others have already suggested, the most likely culprit is a blocked tank vent.
Tank vents have 2 main functions: 1. To provide an escape for air displaced by incoming contents... and 2. Provide a source of air to replace contents as they're pulled out by a pumpout or overboard discharge pump.
Do NOT use the toilet again or try to pump out or dump the tank until you've cleared the vent. Without a source of air to replace contents as they're being sucked out of the tank, the pump will pull a vacuum that'll prevent it from pulling out more than a gallon or two. A particularly strong pumpout can even implode a tank.
The two most common locations for a vent blockage are the vent thru-hull and the other end of the vent line--that end of the hose and the vent fitting on the tank. Start by cleaning out the thru-hull...use a screwdriver blade, ice pick--whatever works. If that doesn't result in a spew out the vent, you'll need to relieve the pressure before removing the vent line from the tank to clean them out...so open the deck pumpout fitting VERY CAREFULLY with a hose at the ready. Be sure you're UPwind of it! Scrape out that end of the vent line and the vent fitting on the tank...replace the vent line.
You'd know all this if you'd read my book! (see link in my signature, just click on the title) The title (my publisher's idea) is a bit misleading...'cuz although it does deal with every source of odor on a boat and how to cure, or better yet PREVENT 'em, it's actually a comprehensive "marine toilets and sanitation systems 101" manual that explains the laws, describes all the types of systems and how they work, and will help you learn how to operate and maintain your system to prevent 99% of problems instead of having to cure 'em. 'Cuz you get to do any preventive maintenance on your terms when it's convenient...the need to cure a problem never happens when it is! And I'm always glad to answer any questions it doesn't.
--Peggie