epoxy over polyester with a gel coat finish is said to be too soft.
Gelcoat is pretty brittle and will crack easily, nothing should go over gelcoat, but more gelcoat and in thin layers.
Epoxy is fine over polyester resin if prepared properly, i.e, clean, sand to 80 grit.
Gelcoat over epoxy works, if the epoxy is fully cured and the amine blush is removed and the surface sanded with 80 grit.
TotalBoat's parent company, Jamestown Distributors, does carry Vinylester resin, although it appears to available only in the gallon size.