If it's still sludge that hasn't turned into "concrete," you can flush it out of the tank by putting 3-4 gallons of water into it via the deck pumpout fitting--'cuz that sends the water into the tank at the bottom to stir up the sludge and hold it in suspension so it can be pumped out...pump that out. Repeat...repeat...repeat...till you're only seeing clean water in the pumpout hose sight glass.
If you're not seeing "mud" in the sight glass even the first time, it means one of two things: 1.whatever is in the tank has turned to "concrete"...or 2. There isn't any sludge in the tank...highly unlikely, but it shouldn't be more than an inch or two, certainly not enough to account for 5+ gallons. Most likely it means that any pickup tube inside the tank was cut too short or any discharge fitting on the side or end of the tank was installed too high. So when the level in the tank drops below the end of the pickup tube, or below the top of the discharge hose fitting at the bottom of the tank, suction is broken...preventing the pumpout from pulling out any more. The pickup tube should touch the bottom of the tank and cut at about a 25 degree angle...a discharge fitting at the bottom should only leave about an inch in the tank.
As I typed I thought of one more possibility: if the discharge is at the bottom and the tank isn't level, but tipped to put the discharge fitting at the high end, the pumpout wouldn't be able to get any more once the level drops to the top of the fitting.
If you do have several inches of "concrete" in the tank, I ran across a product at the Strictly Sail show in California this past weekend called "Tank Tech" which the mfr claims will dissolve it. You can read all about it on their website TankTechsRx.com They're just starting to break into the marine market with it, have been the RV market. I'm not endorsing it and will be checking it out with my own expert sources. Meanwhile, it's not expensive, so I don't think you have much to lose by give it a try.