It's sounding like the tank vent is still blocked...which would also account for what seems like a clogged toilet. 'Cuz if air in the tank displaced by incoming waste can't escape out the vent, the tank becomes pressurized...the toilet becomes harder and harder to pump and flushes back up. That will also cause a pumpout or macerator pump to pull a vacuum that won't allow it to pull out more than about a gallon.
Vent blockages, especially on sailboats, typically occur in two locations--the vent thru-hull and the other end of the vent line...both that end of the hose and the vent fitting on the tank. So if you only cleaned out the tank vent thru-hull, you still need to clean out the other end. Open the deck pumpout cap to relieve any pressure before removing the vent line from the fitting.
If that doesn't solve both problems, we may need to arrange a Skype call to explore all this in more detail than is practical in a forum. If so, send me a PM and we'll go from there.