Having 150-300 lbs. of water at the bottom of the boat near the center of gravity would seem help with stability. If one keeps it empty, does it impact the performance of the boat
Yes, it affects performance. Exactly how and whether that is positive or negative can vary. As JS said,
But...
Some boats steer better with a little more forward trim, others get very tender with empty fuel or water tanks. The designer will consider the intended purpose of the vessel and attempt to optimize for that. A racer is going to be light, light light light. A bluewater cruiser will be equipped to cross oceans and weather storms and house people who live aboard. A weekender has its own criteria.
The hull shapes, the location and amount of ballast the size of the rig all balance to give what the designer believes is the best solution. My Marine 19 is said to perform better with a little weight added forward. Those who have put batteries between the v-berths swear they sail better. A larger boat may not notice the difference of 150 lbs of water or even 300 lbs. If you're under 40 feet, it's worth thinking carefully how you load your gear for a race or an ocean voyage.
Xstream, it would be a fun and informative exercise to pay attention and sail under both full and empty tanks. Does she get more tender, does she get more weather or lee helm, does she track and/or tack and respond differently to jib or genny or shoot, how does she get when the sea state picks up? Take her out and note these things down in your log.
-Will