To reduce heel, learn to reduce sail and reduce draft. If you don't do those two things, then you have to steer off the wind to keep the sails on a reach or a run to reduce heel. Boats heel when we sail to windward ( beat). We can either NOT sail to windward, or we can reduce sail or reduce sail draft to reduce heel. To reduce sail draft, haul the outhaul on the main, haul your mainsheet while balancing it with your traveler, and pull your genoa cars toward the stern. That will tighten the foot of your sails by hauling them aft, and it will draw against the center of the sails. A shallower draft will reduce the lift produced by the sail and reduce the heel in the boat. Reefing is reducing sail. It should be self explanatory. Less sail also means less lift. Not sailing to windward is incredibly inconvenient when you are against a lee shore or your destination is to windward! So learn to reduce sail by reefing, or reduce lift by making the draft shallower.
You really cannot simply let the sails out further while maintaining the same tack unless you let them luff in the wind or balance them with the traveler, which is not good for them. Haul your sheets til the sails are just past luffing, or ease your sheets til your sails begin to luff and then haul them slightly til they stop luffing. That is the correct set. There is more to it in terms of tightening the leech, the luff, the foot, etc to get the perfect set, but start by reducing draft or reefing the sails to reduce heel.
Learn to watch the tell-tale strings/tapes on your sails. You probably have one or more tell-tale tapes on the leech of your main. They will act in 3 ways: 1. Trail straight back off the leech of the sail. That means the air pressure between the windward and leeward sides of the sail is balanced. This is optimal. 2. Dangle or swing forward on the windward side. This means the air pressure on the leeward side is greater than the windward side. Your sail is close to luffing or is luffing. You are stalling the windward side of the sail. Your sheet is probably eased too far! 3. Dangle or swing forward on the leeward side. This means the air pressure on the windward side is greater than the leeward side. Your sail is probably hauled too tight, and you are stalling the leeward side.
You also probably have tell-tale yarns near the luff of your genoa. They do the same thing as I described in 1, 2, and 3 above. They should both be trailing aft to indicate proper balance.
If your main has more than one tell-tale on the leech, start by getting the lowest one right, and then you will adjust the balance between the mainsheet, traveler, boom vang, and cunningham to help shape the twist of the sail as you move upward, away from the foot. Worry about getting one right at first. The rest are more about twist and getting all the energy from your sail, which you should do no matter how much sail you have out or draft you have in them.
When adjusting your traveler: move it to leeward when sailing down wind, and move it to windward when sailing upwind. Moving it to windward when sailing upwind will allow you to haul the boom over the center of the boat and thus point higher into the wind. Hauling the mainsheet in this instance will also create a shallower draft. Easing the traveler to lee when sailing down wind will allow the boom to swing wide, and easing the mainsheet will allow a deeper draft too. There are sailors who never change the traveler! This is not right. They are slow! Many boats have travelers that seem like an afterthought. That is unfortunate. They give up a huge performance advantage to boats with effective travelers, on all points of sail.
The positioning of your mainsheet on the boom will also effect sail shape: midboom sheets will draw against the center more and effect the draft more. Endboom sheets will flatten the leech more.
So you see, you can do a lot to reduce heel while sailing the course you want with good power! Yes, some boats want to heel more according to the depth of their keel and center of gravity.
Hope this helps!
Thanks,
Andrew