I wouldn't worry too much about that. If anything, you would want to rough up these surfaces with a Dremel Tool sanding drum and clean it good with denatured alcohol so that the caulking will stick.
This past spring I had a tiny leak coming in around the frame and lens of my Gray Marine forward hatch and I wasn't sure what to use to adhere it. Rudy at D&R Marine had sold me a tube of Dow Corning 795 a few years ago and he told me that this is the stuff that most boat builders use for plastics such as hatches and port lights. Outside of the stuff being very messy to work with, I had little success with it over a period of a few years. I was forever touching up this lens after discovering tiny leaks.
Then I talked to a cabinet maker friend of mine who owns an O'Day 30 and he turned me on to Dap Silicone Rubber Adhesive Caulking. He told me that he had accidentally broke the lens on his forward hatch and replaced it using the Dap to adhere it.
He told me to rough up the surface, which I did outside on one of the YC picnic tables one day. The boat was sitting at the dock and I had removed the hatch cover to work on it. He had told me to clean the surfaces good so that the caulking would stick and I did.
I realize that it's too soon to really know what the results are going to be because time will tell. All I can say is, the lens is adhered very securely and as tight as the day I installed it.
I've read Don Casey's books and he claims that Silicone does not stick and if anything, it acts more like a gasket and this may be true.
My friend told me to make sure to buy the Silicone Rubber and not the Silicone. Then he mentioned the brand that he used on his boat as being Dap.
If I ever need to remove my port lights again I'm going to use Dap.