My .02 with a trailerable sailboat sometimes used similar..
I think what you said in your first post with the 100 to 200 ah li battery and the 20 watt panel and your overall small energy usage is not that far off for going out four days. Given what you said about power needs, I would guess you will maybe use 20 to 25 ah per day max if you also bring along something to entertain yourself like a laptop. Ive tried 10W, 20W, 30W, 40W with similar loads and even longer trips, 10 and 20 watts means you have to be a little energy frugal and monitoring charge usage becomes more important, 40 watt is somewhat "luxurious" power wise. With what you mentioned, you dont need 100 watt. But as someone suggested, anyone who has an overkill of solar never has an issue with it and the excess charge can be easier on batteries since the discharge is lower. If you just dont want to monitor things, go with larger. If you enjoy learning what different loads do and monitoring things, you can get by with the smaller panels.
My suggestions..
Focus on a panel location that gets very little partial shading. Example, mounting something under the boom might mean you need double or more the panel size compared to mounting the panel off the rear pulpit where it gets very good sun. If you get good sun, 40 or 50 watts is likely a little into the overkill range. Put the panel is a poor location and you may need significantly larger panels.
Get some sort of instrument to monitor your amps, amp hours, voltage etc in and out of the battery. These are usually called Charge monitors. This will really give you a good idea of what you are using in real time and over the length of the trip. When you buy a solar charge controller, get one that will link to your phone and allow you to monitor at least what the solar is putting out. The charge monitor tells what goes in and out of the battery, the solar monitor tells what the panels generated. All good info.
Focus on lower power. Example, if you have a car/marine stereo, get something with a class D amplifier. Use a direct 12V to laptop power brick if using a laptop instead of wasting a bunch of power with the intermediate 110V inverter step.
You mentioned all LED lights - good.
You were probably doing this no matter what but get a solar charge controller that allows for future solar expansion. For example, if you ever did decide you wanted an efficient fridge, you would need an additional 100 to 140 more watt panel. Easy to find a good quality charge controller like Victron that does this.