The amount of power used to start a small diesel is trivial, an amp hour or less. However, that power needs to be there in a split second.
The symptoms you describe, won't start unless the battery is freshly charged and starter having a hard time turning over are symptoms of a poor circuit. Which could mean, poor connections, inadequate wiring, failing battery.
The solution could be as simple as getting rid of wing nuts on the battery terminal (if you still have them) or cleaning and tightening connections to as complex as rewiring the entire starter circuit.
How old is the battery? What kind of battery is it? How is it charged?
How is the battery cable connected to the starter? Are there additional connections on the starter, alternator lead, solenoid lead, etc? Make sure the battery cable is at the bottom connection stack so it is in direct contact with the starter.
FWIW my 1 year old start/reserve battery is exhibiting the same behavior. It won't start the motor unless it is fresh off a charging source.