Match the luff 'hollow' already cut into the sail
There are TWO answers to this question .....First, the maximum amount of sag depends of what you are trying to do! Down or off-wind you can put as much sag into the rig as you want; but, just less than that which would make the sail unstable (too much ‘roundness’ at the luff). Upwind, the amount of sag depends on how high you want or need to point .... The less sag the higher the boat will point .... up to a point at which the shape of the sail becomes too flat in the middle portion of the ‘entry’ sections ( the rounded area right behind the luff ). Second, and perhaps more important is: how much luff ‘hollow’ DID the sailmaker originally put into the sail to compensate for 'normal' forestay sag? huh? Yup, every jib or genoa is lofted (cut) so that the typical normal amount of luff/stay sag for that particular type and model of boat is 'compensated' for in the design of the sail’s luff. Such calculations consider or assume that the rigging is tensioned at 12-15% of its normal breaking (yield) strength and the sail is or becomes windloaded at or near its design maximum wind range at ~17-18 kts. So, if the tension in the forestay approaches 12-15%, and the boat is being sailed in 17-18 kts., the sag that you will encounter will be compensated by the design of the sail. If you want to know what the designed ‘luff hollow’ in your sail is – lay the sail on a flat floor, make ONE accordian pleat/fold about 1-1/2 to 2 feet back from the luff and such that the sail is entirely FLAT on the floor; and then, sight along the luff or measure the difference between a string, etc. pulled tight between the tack and head .... and measure, .... use this dimension as your ‘target’ luff sag. Then, if you need to point higher - add tension to the backstay (which reacts with the forestay) and the luff will straighten out a bit. How much is too much, when you get up to above 30-50%% rig tension ... somethings gonna 'go' soon! Note: If you overtension the jib sheets (pulling super-tight) you will ADD more sag to the forestay ... and will need to add in more backstay tension to compensate. So, how much sag? .... match the amount of 'hollow' that the sailmaker originally cut into the luff is the answer.