If you are connecting and switching line-level audio signals, as coming out of a pair of phono (aka 'RCA') jacks or a stereo 3.5mm jack, then yes you can safely tie all the grounds together and just switch the "hot' audio conductors. You may still get a loud pop or a click when switching if the volume is up.
Ditto for the 3.5mm headphone out of personal portables, iPods etc. Some of these would be happiest if there was a 100 or so ohm resistor across their output to simulate a modern headphone load.
(I was a broadcast/recording tech for 20+ years)
Transmitterdan, and Kenn have it right.
So you know, ground loops are an issue on the voltage supply side, usually due to either different source voltage or different ground potential between connected components. The supply voltage difference creates a current on the signal conductor, (RCA cable) that shows up as noise.
When using the headphone out on a personal audio device, you will see that the 3.5mm plug has three conductors, tip, ring, and sleeve. The aft-most metal portion of the plug, (sleeve) is a common ground, (neutral) for both left and right channels. The AC signals for the left and right audio are carried on the tip and the ring.
Note: audio is AC, so the voltage on the tip and ring swing both positive and negative; this means that the ground conductor sits in the middle of the sinusoidal audio waveform, from a voltage standpoint.
In a kit-bashed switch box, one can tie all "sleeve" conductors together in a hard-wired fashion. Then for any switching, you want a double-pole switch. For each position you will connect both the tip and the ring , (usually red and either black or white inside the sheath) to their respective connections such the both the left and right outputs come out of the switch from their respective poles. The switch prevents "MUX" summing, (paralleling) of the input sources, so you don't have the issues with parallel inputs as mentioned above.
There is some confusion in this thread as to what ground/neutral wires are appropriate to tie together. For the sake of this discussion, and limited to this discussion let me put these points out there.
Small portable devices
*If it is a line-out or headphone out from a small portable device, tying the ground/neutral wires from both channels is appropriate, as it is common and expected.
*if you want monaural sound, tying the left and right wires together is appropriate. This is ONLY the case with headphone outputs or line outputs.
*small portable devices use 3-conductor cables for stereo reproduction. If you see four conductors, stop and ask why. Most smart phones use a fourth conductor for talk-back mic but that is probably irrelevant to this conversation. FYI
12V radio speaker outputs
*It is never appropriate to parallel wire left and right 4-ohm speaker outputs from any mobile stereo. Not the positives, not the grounds. Keep all four conductors separate. Note: an isolation transformer gets you around this but that discussion is for the fourth grade class, not the first grade class...
Now back to the short answer...
Kenn got it right...
