It's all about balance
First thing, you probably shouldn't use the inverter when trying to charge the batteries. Find out how many amps your inverter is sucking down while operating, and compare that to how many amps your alternator is pumping out. Also, take into account anything else running off the DC circuits while "charging". Engine alternators are not famous for cranking out a lot of power. In the grand scheme of things it's almost more like a trickle charge.
Consider a car and its charging system. I'll bet you'll find that the majority of people who have problems with the battery starting the car are people who drive primarily at night when the load of headlights, and other drain sources are being utilized. Yes, it's true, on a car, even driving at night, there is a continuous charge from the engine, but take into account the drain being put on the system at the same time. Detroit doesn't really take this into account when designing systems.
Let's say (only as an example) that your battery is down about half way. Your alternator puts out let's say 90 amps. Further, let's say your inverter is sucking up 40 amps, you also have instruments, radar, refrigeration, etc. totaling another 40 amps. Guess what? You only have 10 amps going towards charging the battery. For an average marine battery I think you're looking at running your engine for 12 hours or more, and even then you might not have a full charge.
I would suggest a full day and night of shore power charge without the inverter running, then try again with a fully charged battery.