You're getting a resonating frequency as air passes around the mast, I think.
The malady has a name, but it escapes me at the moment.
Maybe wrap a line around it? Like an automobile's antenna, that should interrupt the resonance.
Its what makes musical instruments sound their notes .... just a harmonc frequency response from (in your case with an in-mast furler system):
1. the mast is vibrating at the same frequency of the air (vortexes) blowing past, etc. the mast.
2. that BIG open slot at the back of the mast is acting like a 'FLUTE' when air at a precise speed blowing across it at a certain angle.
For case #1 - The remedy for casy of vibration of the mast .... BUT you dont have a backstay ... so one possibility is to tie a 'rope' between the two cap shrouds and tighten down on them ... OR raise a 'heavy weight' aloft thats in firm contact with mast.
OR as others have noted, wrap a halyard around the mast from top to bottom in close 'spiral' to break up the "voretices" (swirling globs of wind) that are shedding from the mast. One or more of these will usually relieve the 'problem'.
However and for case #2, if the 'note' that your mast is producing is because that BIG open 'slot' is acting like a gigantic FLUTE .... youve got to somehow close down that slot opening.
The best way, I envision, to do this 'slot closure' is to get two (cheap) 3/8" or 1/2" dia. 'ropes' tied together side by side ....and with sails slug tied in between the ropes and so tied as to keep the ropes as close to that slot as possible and with the slugs spaced apart about every 36" ... and then raise 'the contraption' like you would when raising a 'standard' mainsail. Without the slugs, any halyard that goes straight up along the slot will whip and slap and can rub the anodizing off the mast. The contraption would go on the outside of the mast.
That you dont notice such vibrations when your mainsail is inside, would tend to indicate that your mast has become a FLUTE so perhaps you should attempt the "hindu rope trick with sail slugs', etc. as a first attempt.