Your description suggests that the leech timing between the two sails is 'off' - the upper panels of the genoas leech are too close to mainsail 'up high'. The genoa was cut with insufficient 'shoulder' (fullness) up high or the sail isnt being correctly shaped when raised - minor/normal inconsistencies when the sail was put together. Several options for analysis & correction ... but includes the installation of 'complete' set of tell tales to visually examine whats happening.
If the top of the main is luffing (backwinding), this is usually a 'slot problem' and not necessarily a mainsail shape problem but usually a shape problem of the genoa. Backwinding is NOT air hitting the back of the mainsail whats really happening is there isnt enough FORWARD component of air flow being generated on the windward side of the main and because of this the jib/genoa isnt providing sufficient leeside flow !!!! Aerodynamics isnt 'intuitive'. Small bubbles of luffing on the mainsail luff especially lower down are usually of NO consequence, in fact they mean that the air flow 'around' the slot is near 'perfect'.
Suggested course of analysis:
1. Evaluate the mainsail for correct set and raised shape:
http://forums.sbo.sailboatowners.com/showthread.php?t=120970
2. Suggest a
full set of tell tales on all sails and a set of steering tell tales on the genoa:
http://www.ftp.tognews.com/Publications/Arvel Gentry Articles/ ..... articles 08, 09, 10 and 11 (in that order) this will start to get that timing of the leeches and their exit shape correct.
3. Backstay tension:
http://www.ftp.tognews.com/GoogleFiles/Matching Luff Hollow.pdf This will correct MANY as raised problems and you may need to increase the genoas halyard tension to 'trip' the leech or cause the leech exit or aft section to become "more parallel" to the boat's centerline when beating, especially near the mainsails second from the top batten. You may temporarily jam a reefing furler if you apply strong halyard tension .... but this will only be temporary and you can always loosen the halyard when you need to reef/furl.
If the backstay is too loose the forestay will be sagging off to leeward and this causes the genoa to go draft aft and with a hooked up leech (closing down the slot open distance) .... a hooked up jib/genoa leech especially 'up high' will be geometrically closer to the mainsail than one that has a flat leech exit shape. More backstay tension and/or more halyard tension will 'open' the jib/genoa leech and will cause the draft to move forward in the sail. The key here is that the aft section of the genoa leech should have essentially the same relative top to bottom 'curve' as the leech of the main and the leech of the genoa/jib shouldnt be 'hooked up to weather/slot' and the tell tales on the genoa leech should be quietly streaming 'straight back'. If the leech is hooked up the leech tales will be 'twirling' off to leeward.
Gorillas on the jib sheet winches can easily overpressure a sail and cause the leech to 'hook up' ... easy on the sheets !!!!!!!! If you need to pull the clew inboard for better pointing, consider to use a barberhauler instead of putting extra strain on the winches. For light winds the barberhauler and comes in and the genoa leech should be closer to the main; for heavier winds the clew goes 'out' either by looser jib sheets or easing up on the barberhauler, etc. .... to preserve that leech 'timing'.
If you need more draft forward to effect a more open genoa leech, then sometimes you need to FLATTEN the mainsail luff section ... as long as the helm balance is under reasonable control:
1. slightly loosen mainsail halyard tension to get the less 'rounded' luff shape but not too much loosening so that weather helm noticeably increases, followed by 2. tighten up on Genoa halyard tension. --- Careful here. only loosen mainsail halyard tension enough but not too much as to cause the mainsail leech to hook up to weather - increased 'weather helm' and the boat will now heel over aggressively instead of being 'fast'.
This a LOT to consider here, but take your time and verify the initial as raised 'set' of each sail, follow the 'tell tale' advice in those articles ... get those leeches 'timed' and perfectly aligned parallel to the boats centerline - especially in the mainsail at near the second from the top batten and the at same adjacent 'altitude' for the genoa. Keep the gorillas off of the jib sheet winches until the sail 'breaks in' and takes its final 'as designed' shape. Just remember its not how much a boat is heeling over but how each correction allows the boat to go 'faster' when pointing .... all corrections should be considered versus the highest VALUE you can get out of the
speedometer (or VMG indicator if you have one).
Rx: Leech timing, slot open distance, and FLAT leeches especially on the genoa !!!!!!!
If the above doesnt affect satisfactory correction, then think about stiffer battens @ #1 & #2 but with less batten compression for less mainsail draft 'up high'.
