A couple things
We need to agree on the function of the sail controls and terms being discussed here.The traveler is used to adjust the angle of attack of the main (within the span of the traveler beam) once the desired twist of the main has been achieved. Moving the traveler car should NOT affect the twist (downward pull) however, yours will, given how it's reaved (more in a minute.)The main sheet is used to provide the downward pull (within the span of the traveler beam) which determines the twist of the main. Once the boom clears the end of the traveler beam a vang takes over the twist control and the mainsheet now only controls angle of attack.We also need to be sure we're all discussing the same thing when we talk about 'flattening' a sail. Hauling in on the mainsheet will indeed make it less twisted (think vertical curve) so that it is (in a sense) 'flatter.' However, most experienced mainsail trimmers refer to the flattening of the main as the reduction or relocation of the camber (think horizontal curve) of the sail. Typically when we discuss the flattening or the fattening of the sail we're referring to the thickness and/or shape of the cross section of the foil.Tightening the backstay tension on most stout masthead cruising rigs will not bend the mast enough to relocate the main draft. That usually only works effectively on bendy fractional rigs. It will flatten the headsail by tensioning the headstay however. You can flatten (reduce draft) on the main using outhaul and halyard/cunningham adjustments.The way your mainsheet is terminated and reaved to port and starboard will probably hamper the traveler's ability to move the boom athwartships when the mainsheet is tensioned. In other words, your mainsheet will always be partially tensioning your boom to the center regardless of the position of the traveler car.My solution would be to terminate the mainsheet on a becket on either the top or bottom mainsheet block then exiting the block sets forward to the base of the mast, out a deck organizer/turning block then back under your dodger. This will put your mainsheet tension entirely within the vertical plane of the boom and allow the traveler to do what it's designed to do.I'd suggest reading everything you can get your hands on regarding sail trim and maybe a visit to the Garhauer website.Hope this helps.