Shunts?
Chuck:
While Rick is right, and we need much more information, here are some general thoughts.
Since the voltmeter appears to read well and the amp meter does not, the problem is probably the wiring of the shunts. This is because the voltage will appear without any current flowing, but as soon as the current starts there is a voltage drop somewhere in the circuit, and the fact that the meter reading fluctuates says that the current fluctuates (I am assuming the meters are OK because they are new and therefore reading correctly the fluctuating current). So what you need to do is to find out why the current is fluctuating.
The connections at both ends of the shunts must be very clean and must have contact with the full cross section of the wires they connect to. Since your meters are designed for 500 amps, which is a huge amount of current, I suspect the shunts are inserted into the battery cables. The shunts should be as close to the batteries as possible--the length of the wiring from the shunts to the meters is not important, as those wires carry very little current. The usual arrangement calls for lug connectors to be attached to the battery cables, which are then fastened by tightening a nut or a bolt, squeezing the lug between two washers. The lugs should be crimped or soldered to the cables, but it's best to buy new cables of the right lengths.
There is also the possibility that the battery cables have poor connections to the batteries, or to wherever they connect to--master switch, engine ground, etc. You have to look at the entire circuit.
So unless the meters are defective, you just have to proceed methodically, sleuthing out bad connections.