First...Welcome aboard sailboatowners.com!
Sounds like your holding tank vent is blocked. All tank vents have two purposes: to provide an escape for air displaced by incoming contents and to provide a source of air to replace contents as they're pulled out. When no air can come into the tank via the vent, the pumpout pulls a vacuum that won't allow more than about a gallon to be pulled out.
First, don't use the toilet again until the vent is clear...flushing against a blocked vent pressurizes the tank, which can have some very unpleasant consequences, the least of which is a geyser out the pumpout fitting when the cap comes off (you don't want to know what the worst can be).
So first lets make sure none of those levers are causing the problem. If there's one in the pumpout hose, it's there to provide a choice of pumping out or dumping the tank at sea. So you need to make sure that one is open to the deck pumpout fitting...the handle should be aligned with that hose, not crosswise.
If it's not the y-valve alignment, it's the vent... And most holding tank vent blockages occur in the vent thru-hull. So use a screwdriver blade, ice pick--whatever works to scrape it out. If you still can't pump out, there's a blockage at the other end of the vent line--the connection to the tank. Open the deck pumpout fitting to relieve any pressure, then remove the vent line from the tank and scrape out that end of the hose and the vent fitting on the tank.
That's the core of "holding tank 101"...if you need more help, send me a PM (private message) and I'll help you figure out the system.
--Peggie