If I'm understanding your description correctly, a tight mainsheet and boom vang should essentially take all of the load off the rearstay when the boom's pulled in tight to centre-line during a closehaul...? In my case I was sheeted in tightly on the main when I noticed the otherwise tight forestay begin to sag during puffs and had the vang on too. I previously thought that the vang had more to do with controlling boom height while reaching downwind when there is more mainsheet out... So much to learn!
The backstay is about the mast and forestay sag, so assuming the sail's leech is a viable substitute for it will quickly get you out of sorts with mainsail trim. Therefore, I suggest you reduce the mounting confusion by going back to your initial understanding....
Upwind the mainsheet controls leech tension, aka twist. The traveler controls the angle of attack, or the direction the mainsail points into the apparent wind direction. The vang, especially on a loose foot main is not especially needed on these points of sail, but it can be firmed up to reduce rattling sounds.
Downwind As the boom eases outside the range of the traveler's vertical control, the mainsail takes over the angle of attack function... and the vang is used to maintain the desired leech tension... essentially to keep the boom from bouncing. Can you visualize that in this case neither the mainsail leech, vang, mainsheet, etc would have any effect on the backstay...so clear that idea from your head. The only reason it was mentioned is to suggest the mast would be stable even if the backstay were floppy... because the sail would help keep it upright... as long as your're going upwind, of course. Downwind, with the boom swung out wide... who knows.. but your boat has swept back shrouds also, offering more fore and aft support.
One quick question before I try to retackle the rig tuning: to readjust the rake do I first need to de-tension EVERYTHING (including all of the shrouds) as well? Also, should it be rigged with the added weight of the boom/mainsail/topping lift attached? I'm assuming so, but want to be sure to do it right.
Try the forestay by itself at first, with the plumb line rigged with mast and boom set up. You'll probably need to ease the swept back shrouds and the backstay. If the Cap shrouds (uppers) are abeam, not swept back) you will most likely not need to change them. The baby stay will most likely show some slack, so you'll want to address that once you've got your rake set. Listen.... All you're doing here is adjusting the fore and aft position of the mast. Which means that if you make an adjustment to the shrouds you need to make them equal on each side. ex. two turns port, two turns starboard... Keeping the tension equal is more important than the amount of tension.... because that's what keeps the mast straight. Once everything else is set up you can go back and tweak the shrouds... there are many good mast tuning guides on line... basic for all sailboats... try the one at
www.cncphotoalbum.com or any spar maker's website.
To what Joe wrote.