I agree with Jackdaw and many of the other posts here, especially those about the size of a 150 jib.
I have an O'day 23 and it gets interesting in winds above 20 kts.
I do not race my boat but do strive to get the best out of her every time I hoist the sails.
I think a 150 is too big in winds above about 15 kts for a center-board type boat like ours where the major keel weight is close to the waterline. As you have noticed the boat will round up especially in winds greater than 15 kts with a 150. I have a 150 and it makes the boat point higher and go well in steady winds up to about 12 knots with puffs up to about 15. For winds greater than that the boat becomes a chore to handle rounding up etc etc.
About racing ... I have crewed in a number of boats for more than 20 years and my experience is that flying the telltales and keeping the main adjusted is the first step to doing well. The info by Don G on this web site is excellent as are the simpler sail adjustment sights which talk about the use of tell tales.
New sails help but getting what you have to "Sail well" is critical. There are lots of adjustments which can be made, - weight distribution of the crew, halyard tension, position of the cars, traveler position, boom position, Cunningham on the main, outhaul, leach lines on the jib, to name just a few. All of these make small differences but the first step is to drive a steady course and keep the power on. Rounding up is like putting a brake on. Having too much sail up and healing the boat is another issue Yeah it looks good and hearing the water rush along the rail is fun but it does nothing for boat speed or the ability to point towards the mark - bigger boats with significant keel depths and weights can carry a larger heel but the O'Days perform better - in my opinion - if you keep them on their feet by tucking a reef in for the upwind leg. When reefed the boat will sit better in the water, there will be less rounding up and they will sail faster, at least that is my experience.
The next time you are out enjoying a cruise in winds greater than say 15 and the boat rounds up, take a look at the stern and the position of the rudder and you will see the water flow showing you that the rudder is a brake and with a huge amount if water turbulence behind. This is just drag and now you have to get the boat going again. Try putting a tuck into the main and see how the boat sails.
If you have a speed indicator you can check speed improvements for the changes you make.
Either way its all just fun and everyday is a chance to improve your skills.
Fairs winds and keep us informed of your progress.
James