If racing handicap, you might check on where the rating break is for headsail size. Sometimes you might find (like a friend did locally) that the better range of something smaller like a 128 benefits both boat and crew.
Also, the J-22 (if I can believe the internet...) was introduced in 1983. This was in an era when
everyone was encouraged to have a 150 or larger. Partly for sailing, and then it sort of morphed into a "manliness" sort of thing.
(sigh)
Several years ago a sailmaker friend advised me to consider moving down a bit from our 135, which was overpowering us some in winds over 15 kts.
Similar to your boat design, our boat is not only fast but very easily driven. Since we do not race anymore, a new composite 95 with three battens (and that's important, btw) was built for us by North.
While we are slower in TW under 8 kts, above that to the mid 20's it's great. Off wind with that jib we can still do 10 plus.
Given that you are racing a lot and perhaps less often day-sailing, you do need the best combo for speed that can drive you thru the puffs. Anyhow, do consider your options carefully, as correct headsail sizing is sometimes more than just making the numbers larger.
And, apropos of whatever, get yourself a GoPro and post up some sailing video, like Jack!