First.. it is very dangerous to mess around with the propeller when the transmission is in gear because a diesel can fire or just pop when you turn the engine over..
before trying this please think carefully about what you are doing.. first, fire the engine up for a second and kill it with the fuel cut-off.. leave the cut-off pulled and do not push it back in until you are finished. try to start it up again (5-10 seconds) to make sure that the cutoff is working..if the engine pops, do not go further with test. after you've proven that the fuel cut-off is working, get a strap wrench and a torque wrench. The strap wrench should fit the engine front pulley. The torque wrench should fit the propeller nut. Pull the cotter pin from the prop nut and put the torque wrench socket on the big prop retaining nut. Put the transmission in forward .. have friend hold the pulley with the strap wrench. turn the torque wrench counterclockwise (opposite normal rotation) .. The prop will turn a little, maybe half to one turn, then it should lock up.. you should be able to get to about 50 foot pounds on the torque wrench.. if the nut backs off below 50, tighten back to about 60, blocking the propeller or put the transmission in reverse and have the friend hold the pulley while you tighten.. then go back to forward gear and counterclockwise measurement.. These numbers are about twice engine torque because of the transmission ratio.. Note that the transmission will slip when the shaft is turned in the "correct" direction.. unlike an automobile transmission.
Again, this is dangerous ! If the engine pops and a human part is in the way, that part will be removed or damaged ! I am not recommending the method, just answering your question.
A better method is to use a digital hand-held tachometer.. With the boat running at a pretty good clip in smooth water, measure the rpm at the front pulley, then immediately measure the rpm (with the same tachometer) at the output coupling.. divide small into big and you should get the transmission ratio for YOUR box..which is stamped onto the plate on the side..