I also installed a Tiny Tack D on my YSM12. The YSM 8/12 series is a great motor. Assuming that your propeller and hull are clean and you don’t have any fuel or exhaust elbow issues, I would suggest that you are over propped. The full load speed should be about 3,000RPM on the Tiny Tack (don’t use a hand held tack on the PTO shafts as they are slower). Your propeller is either too large in diameter or your pitch is too large. Some say that one inch of pitch reduction will increase your full load RPM by about 100 RPM and one inch of diameter reduction will increase the speed by 200 RPM.
When I bought my boat four years ago she would attain the proper 3,150 RPM in neutral and would not over heat. In gear she could not get over about 2,600 RPM (tied to the dock or running) and the high temperature alarm would come on after about 5 minutes. Boat speed was up to hull speed (6.5 kts) and the stern wave was climbing up the transom. The previous owner had the mistaken belief that he could push the boat faster in wind, current and waves with a bigger propeller but the only thing he accomplished was to burn more fuel that could only be converted to excess heat. The installed propeller was a 3 blade Campbell Sailor 13 inch diameter x 9 inch pitch. I contacted Campbell Sailor and on their recommendation had it modified to 12 inch diameter and 7 inch pitch. She can now run at the full load 3,000 RPM and a bit over 6 kts all day without overheating. I normally motor sail at 2,700 RPM and about 5 to 6 knots and consume fuel oil at about the rated 2 litres per hour.
When trouble shooting, ask yourself:
What has changed?
What was worked on last?
Has it always been like this?