It would be better to power the frige with a 12v power supply when you aren't there instead of the battery charger. The high load and intermittent demand of the frige can confuse the charger into cooking the batteries. (May not allow the charger to go to a "float" mode or an "off" mode) A good electrical guy could wire one in using a relay, or relays, such that it would switch away from the battery and onto supply when the master 120V panel is switched on. This could be done easily with a manual switch too but you'd have to devise a method of not forgetting to switch over.
EDIT: it won't do anything bad immediately, but over time (maybe 1-2 years) it will significantly shorten our battery life