Control Angle of Attack with the Traveler, then Mainsheet
While the end of the boom is over (or near) the traveler, the traveler controls the angle of the main to the wind. The mainsheet controls twist, allowing the top of the sail to fall off the wind.
Once the end of the boom is downwind of the traveler, the mainsheet controls the angle of attack of the main, while the vang controls twist. If you don't have a vang, once the boom is off the traveler, you cannot separate twist and angle of attack. That is, easing the mainsheet will increase angle of attack, _and_ increase twist.
Don Gillette's guides will tell you how to read your telltales, and optimize your sail with these controls.
Keep at it! Tuning your sails is straightforward, but you do need to know what you are doing, and what each control does for you.