OK. Trace the charging wires from the Alternator. Do they go to the switch? (I suspect this based on what you stated).
If so then the charge from the alternator is distributed to thru the switch to the batteries. MaineSail does a great write up about this and the problems it can create.
There are some issues with this design. You can change this and improve your system.
Simply stated, a better design is to wire the alternator directly to one of the two batteries. Preferred choice would be the one you use to power the gear (GPS etc.) on your boat. Lets call this the "House" battery. Now you need to some how keep the "Starter" battery charged. You use a ACR (Automatic Charging Relay
https://www.bluesea.com/resources/1366 ) to share some of the charge to the Starter battery. It is explained in the link. When you do this then there is no need to use the "Both" function on the switch. Again MaineSail has an in depth review of the 1/2/B switch. Or you can read the info shared by Stu Jackson on this forum. Just search for 1/2/Both and you'll find the detail. The recommendation is to stop using the switch to direct charging. Running your charging circut straight to the batteries. This also helps to assure that your batteries are getting all the charging voltage that is available from your systems. Anytime the charging circuit goes though several connections there is a bit of loss.