The bearing in the rudder tube, at least on my boat, is basically continuous from the bottom to the top with a break where the quadrant attaches to the stock so I wouldn't cut into it if you can avoid it. Here's a picture of the clamp I made to press the pin out, I never got a chance to use it but that's ok with me:
