I don't think the problem is in the toilet...
It's more likely to be downstream of the toilet....something creating back pressure that's pushing your flush back into the bowl.
If there's a y-valve in the toilet discharge line, it may be turned to go directly overboard, pushing against a closed thru-hull. If you can eliminate that possibility, check your tank vent for a blockage. The two most common locations for a vent blockage are the thru-hull and the vent line connection to the tank--both that end of the vent line and the tank vent fitting. If it's a brand new boat, most likely the vent thru-hull is blocked. If you can't just scrape it out with a screwdriver blade, or if that's not the problem either, call your dealer...something is messed up in the plumbing.
Don't try to use the toilet again till you solve the problem.
Operating the toilet should be straightforward. Hold the button that brings in flush water down just long enough to bring the desired amount into bowl...any more than a pint or two is wasting water. The "flush" button should empty the bowl. If it doesn't, something is blocking the discharge. If you need a little more water to rinse the bowl, repeat the process. If it's not working, quit...continuing can create worse problems.