There are a few different formulas, all of which share one important variable. Different symbols for it are used in different formulas and programs but, here, we can call it "F". This stands for "Fudge Factor" and it the primary determinant of the final diameter/pitch combination.
The larger the diameter, the more effecient the prop will be and the less fuel you will burn. However, there is a maximum speed the tips can go through the water without wearing away the metal so you have to keep raising the reduction gear ratio of the transmission. Since you probably aren't going to change transmissions, that is one size limit although it's a bit fuzzy. Tip speed that a yacht with less than 100 hours a year running time could get away with would be a big problem on a commercial vessel operating nearly full time. That formula needs a big "F" in it.
You need to have 10% diameter tip clearance if you are willing to tolerate a fair amount of vibration, 15% if you want it reasonably quiet, and 20% if you don't want to hear any significant propeller noise. Few sailboats can be arranged with the tip clearance for a really quiet prop. Put and "F" in that formula for owner noise and vibration tolerance.
Once you have a diameter, probably determined by the above, you then want to select a pitch such that the combination of the blades going around and the boat's forward speed will cause them to encounter the water at the optimum angle of attack. This means you have to know how fast the boat is going to go. That's a whole other question. Part of getting it right is knowing how effecient the prop is going to be and you don't know that yet.
The big "F" in the formula is the fact that the hull drags some water with it and this slows down the flow past the prop. The water speed at the prop is not the same as the speed of the boat through the water. There are rules of thumb for this based on the fact that fatter and fuller ended boats slow the flow down more than long skinny ones. Still, to get the prop exactly right, you have to know this factor. Without a super computer, the only way to know it exactly is to have data from a very similar vessel. This needs to be accurate trial data and not bar talk.
Most places that can sell you a propeller will have a computer program with enough "F" values established for common boat types that they can sell you a prop you can live with. If you want an exact match and are willing to pay for a day of careful sea trials with exhaust temperature and fuel flow measurements as well as a haul out, you'll probably have to switch props once or twice. This is usually only done on commercial power craft where a 1% fuel savings a year can pay for all that engineering.