Thank you everyone for the information. One thing I already see is that the service manual never mentioned that the cam rotates at half the rate of the crank, though now I'm like "duh" should have thought of that. The manual just says to put the cylinder at TDC by flywheel mark, it never mentions that you have a 50/50 chance of this being right. So basically what I was doing was chasing my tail, because maybe I'd catch the TDC on compression, maybe not, but whichever one it was, I'd set the valves, then rotate the crank 360 degrees back to that cylinder expecting to see the same valve readings, and I was totally flabergasted that they weren't. So now I'm thinking I may have broken something, because here's the rest of the story: Just bought the boat last September. Have put less than 12 hours on the engine in that time. Still working though routine maintenance issues like fluid and filter changes, etc. Coming back in from a sail under power. Notice engine RPM falling off. I'm thinking "plugged fuel filter". Quick email to previous owner reveals primary fuel filter not changed for years. OK. Replace primary - very black. Replace secondary - looks OK. Bleed system. Noticed previously mentioned solenoid problem, which led to my tackling the valves. Decided to test engine thoroughly before taking her out again - terrified of having engine die while transiting narrow channel into/out of Bayou. Started engine - fired up immediately, ran smoothly at idle. Warmed up for five minutes, then increased power to half throttle. Ran for ten minutes. Ran smoothly, only very very light whitish smoke. I thought "great, doing well." Decided to run it up to full throttle for five minutes, then bring it back down. Ran at full throttle for two minutes or so, then engine started chugging and fell off to idle. Since then, it will start, runs somewhat rough, dies when I apply throttle. I figured "last thing you touched is probably the problem" so I re-checked the valves, and they seemed way off. I figured maybe I hadn't tightened one or more tappet lock nuts sufficiently, and they drifted at high revs. I've been chasing this ever since, and have finally given up and called a local diesel mechanic who is highly recommended, and who must be pretty good because he is "booked up for months."
So, some thoughts I'm having. Maybe I got lucky enough at first to get it ALMOST right, so that the engine fired up and appeared to run smoothly. Is it possible that running it that way at full throttle for a couple of minutes could break something? I wouldn't think I could have caused a valve interference problem. One thing a friend of mine who works on diesels has suggested I try (obviously once I'm sure the valves are right) is to put in an electric fuel pump, because he says he's seen diesels with plugged fuel filters overstress the lift pump enough to cause a fuel starvation problem once you get them running again. I don't know, sounds to me like the fact that it started and ran fine for over 15 minutes makes that unlikely, but he says he's seen it and it's an easy thing to check, could explain why I can fire it right up but it dies at any throttle. I have verified that I'm getting fuel at all three injectors. So many things to think about!