Sludge isn't causing your problem
And I think you're referring to your holding tank, not your head...unless you have the largest self-contained system ever installed on a sailboat.

Let's apply some logic to your problem: It would take at least 50 years for 40 gallons of sludge to build up in a 50 gallon tank. It would also gradually reduce the tank capacity as it rose...your tank capacity wouldn't suddenly go from 50 gallons to 10. However, even that scenario is impossible...because the accumulation of sludge would only have to get deep enough to cover the head discharge in the tank to clog the discharge hose up and prevent ANYthing from being pumped out. So there's no way that accumulation of sludge could be the cause of your problem. I'd bet your tank vent is blocked, preventing any more than about 10 gallons from being pumped out before the pump pulls a vacuum....removing the pumpout fitting cap would let just about that much air into the tank. I'd also bet that if you have a manual toilet you've been feeling increasing backpressure when you pump it--sometimes, if not all the time...and that you've even had at least minor eruptions when the fitting cap has been opened shortly after using the toilet.The most likely places for a vent blockage are in the thru-hull and at the connection on the tank. You'll most likely need to scrape both out with a screwdriver blade to clear it. To prevent the vent from becoming clogged in the future, backflush it each time you pump out and/or wash your boat. Although sludge isn't your problem, unless you regularly rinse out your tank, you do have some and it should be flushed out--at least as much of it as possible--once your vent is clear....'cuz sooner or later it WILL get deep enough to clog the pumpout hose. If it's sat through enough winter layups to let any of it turn to "concrete," nothing will dissolve it that won't also damage the tank...but if it's still "mud," it can be pumped out by repeatedly sending a few gallons of water--enough to cover the bottom to about 2"--down the pumpout hose--'cuz that sends the water into the tank at the bottom where it can stir up the sludge so it can be pumped out. Keep doing it till what's being pumped out is clean water. You should at least nominally rinse out your tank after EVERY pumpout...and THROUGHLY flush it out at the end of every season.