An a/c circuit does not need a ground to operate. The current's return path is the neutral wire. Electrical code dictates that a circuit should have a ground wire leading to the ground wire at the power source in case there is a short circuit in an appliance and its case or metal components become energized. In such a case the ground wire would provide a more efficient path to ground than a person coming into contact with the faulty appliance. In a circuit with a ground wire properly connected and working the GFCI is just a redundant fixture. It is still a receptacle with a ground that in addition has a fast power disconnect switch. I just don't think that in a boat a person could that easily become a conduit to ground or let's say as easily as a person could in a home sitting in solid earth. I once tried installing a GFCI receptacle in a older boat and abandoned the project as the box was not deep enough to accommodate the fixture. Decided the existing grounding system was more than adequate given the environment. Don't clearly see how the 12V system could be energized by the ground wire of the ac circuit. The ground wire goes to the shorepower connector and not to the engine ground as the 12V does.
Most of this post is simply incorrect and dangerous information.
It takes as little as 30 milliamps of 120 v AC to kill someone. The typical household and marine GFCIs trip at 5 milliamps, well below the level that will kill someone.
ABYC compliant wiring has both the DC negative and the AC ground connected at a common point. This is to ensure a path to ground for any current that leaks into the DC circuit. Connecting the AC Ground and DC negative also provides a pathway for AC to enter the water, which in freshwater can cause electroshock drowning. A GFCI prevents any residual current resulting from a ground fault from reaching the water because it trips. Current ABYC standards require an ELCI, which functions in the same manner but at a higher trip level, to be installed in the AC system before any branch circuits.
Nuisance trips occur for one of three reasons:
Faulty GFCI
Installation in a wet environment
A fault in the circuit, either in an appliance or in the circuit wiring.
Given the location of the outlet in the OP's boat, the second reason, wet environment, is a likely culprit. If moving the outlet to a drier location doesn't cure the problem, then there is probably a fault somewhere in the wiring or appliance he is using.
GFCIs should be the first outlet in a circuit as they protect not only the outlet but all outlets downstream.
For more information on residual current devices, take a look at a reliable source like the Wikipedia entry.
en.wikipedia.org