Look at the general cleanliness of the engine and surrounding area, should not smell diesel. Check the oil dipstick and look in the valve cover as best you can to determine if the oil was changed regularly (oil will be black but any metal should look clean with no "varnish" or sludge apparent). Make sure the engine coolant looks clean. Look at the exhaust exit for excessive soot or oil residue. Look closely at the raw water pump to determine if it is leaking - look below the raw water pump for signs of rust, especially on the oil pan - if this is a fresh water boat, then doubtful rust will be a problem. Look closely at the heat exchanger for leaks, condition. Look at the motor mounts, likely are original. Check the transmission fluid, should be clean and at the correct level.
Hopefully you are able to run the engine, if so:
* See how easy it starts when cold (should start quickly after running the glow plugs).
* Diesels are noisy - check for "excessive" noise.
* Look at the exhaust, shouldn't be too smoky, check for black (fuel), blue (oil) and white (water) smoke.
* Make sure there is sufficient cooling water coming out of the exhaust (RPM dependent).
* Warm up the motor to operating temperature.
* Run the motor at various rpm's, including full throttle and check for smoothness/vibrations.
* Put the trans in forward, apply some throttle, put in reverse, apply some throttle - have someone ensure the engine doesn't move much when going forward to reverse (motor mounts).
* Make sure the engine temperature has remained consistent.
* If you can take if for a drive, put the motor through its paces.