A Stiffer Transom
Since adding the Starboard backer didn't give you the stiffness you need, you'll probably want to build a sandwich panel to get the stiffness you're looking for. This involves bonding a core (probably marine grade, untreated plywood - I'm not sure how epoxy would bond to Starboard) to the transom then laminating a couple layers of fiberglass to the plywood. In doing this, you don't want to just create this stiff transom independent of the rest of the hull, meaning, once the cored laminate is in place you should create fillets around the outside edges to bond it to the sides of the hull and tape over the fillets to spread the load into the hull.