I don't know your area, but around here there is no place you can go to completely avoid crab/shrimp pots.
We sail at night from time to time. The key word is "sail". Motoring at night is fairly risky, largely due to pot markers.
But, the scarier thing is logs. Every now and then we come across one that is big enough to do some damage. During the day, we are sometimes surprised by a close encounter. At night, it is mostly just a blind gamble.
We prefer to night sail toward some light (city, moon, whatever) so we have some visibility of what's on the water, but the risks are all higher - except maybe risks involving other boats.
There is something special about it if the conditions are right.