It's not too hard...
Don't be dissuaded by the other posts. Replacing a head liner is really well within your capabilities. I restore boats professionally and I have replaced dozens of head liners.
The limiting factors are your creativity, your access to proper tools and your budget. Since you mentioned you're not a carpenter and you don't have a ton of money to throw at this project, let's rule out teak panels, tongue and groove or other forms of wooden head liners.
Economically, you still have some choices. The easiest, cheapest and simplest are foam-backed vinyl, laminate (formica) or going without... You can use plain vinyl without the foam backing, but the foam will hide (or at least soften) surface imperfections. Because you had foam backed vinyl on the boat already, the simplest thing to do is replace it with the same product.
The first thing you need to do is prepare the surface for a new liner. This means more than just dusting it off with a whisk broom. Start with soap water and a scotchbrite pad and remove all traces of the previous liner, foam and as much glue residue as possible. Then (wear gloves - nitrile is good, so are heavy duty kitchen gloves) wipe down the entire contact surface with a solvent. -I recommend denature alcohol. Acetone will make quick work of the glue residue but it's odor is more toxic and it flashes off (evaporates) so quick that you'll be constantly rewetting the surface. Denatured alcohol actually has fewer impurities than acetone, is more user friendly and won't get you very high.- After wiping the surface down, try not to touch any part of the hull or headliner with your bare skin, this leaves an oily residue that can weaken the bond of the glue you'll use.
If you still have the discarded vinyl you can use them as templates, if you don't, then you'll have to make new panels. It's not necessary to be that exact here, just cut about and inch more than you need on all sides. it will get trimmed down later.
*important* If you are holding up fabric against the hull to get the right shape, use a pen to mark it, then cut it later on a flat workpiece, don't try to cut it in place. At best you will slice up the finished surfaces of your boat and at worst, you'll have to adjust to the nickname, "Lefty".
Once the panels are cut and the substrate is clean, you can apply the contact cement. Do NOT get the waterbased (often marketed as "nonflammable") contact cement. This is what failed you the first time. You want the urethane based, extremely toxic, explosive stuff with warnings all over the can. Did I mention it's extremely toxic and explosive? That's because it's extremely toxic and explosive. I only mention that it's extremely toxic and explosive because of its extreme toxicity and explosivity.
Don't worry...
Buy a respirator (about $30-$40) not a gas mask, not a dust mask, not a handkerchief tied over your nose like Butch cassidy. Get and actual NIOSH approved particulate respirator with cartriges. This will protect you from the EXTREME TOXICITY. Then turn off the power to the boat and extinguish any pilot lights you might have aboard. If you have any friends who are learning how to eat fire or possibly surprise you with a birthday cake, advise them to stay home. This will protect you from the EXPLOSIVITY. Ok, just to say it, because some people really are that stupid, don't smoke.
Open the hatches and spray or brush the contact cement on both the hull and the liner material (yes, the foam side, not the vinyl side. Seriously, I've seen it done). I prefer brushing it on because you can get and even coat and there's less waste, but spraying is fine too. When it comes time to do the vinyl, take it outside and coat it there. Only apply the cement inside if there's no ther choice.
Once it dries (about half an hour at normal temps) do it again. You don't want to have to face this problem again in a year and raw fiberglass is a fairly porous material. Wait for it to dry again (it will be claer and glossy) then CARFULLY, start applying the liner. It will bond instantly so take your time line it up right, start at the middle of the top and work outwards and down. If you screw up and get a wrinkle, try to press it flat rater than pulling around it. The glue will have a tenacious grip and will only cause you to strech the vinyl around the wrinkle making it look like a worse mess than it started out to be.
After you've applied the vinyl cut off (or tuck in) the excess material. You can also judiciously use strips of teak trim to give it a finished look.
For the cabintop (not the hull liner) you can use the same procedure. If, however, you have wiring to hide or need access to deck fittings, you will want to use panels.
For panels, I recommend laminate (Formica), but I'll tell you how to do vinyl also (laminates are great for cabintop headliners, but lousy for hull liners because hulls have compound curves and laminates will only curve on one plane).
First, you'll need something for the panels to attach to. If you're not handy with tools, purchase some strips of wood (3/16" to 1/4" will be fine) that are a little longer than the width of the cabintop. The camber (or curve) of the cabintop will suck in some of the length. Use tape or a pencil to mark exactly where you want the panels to be. They can be any size or shape you want, but anything bigger than about 3' x 5' becomes unwieldy to work with. Generally, panels run in about 2' widths all the way across the cabintop. This is the easiest and most visually appealing method.
You're going to laminate two strips together making them a total of about a 1/2" thick, but you're going to start by just doing the first piece of each furring strip. Any time the strip must cross wiring or anything else you may need access to later, cut the strip leaving a gap for the wiring (the second layer of the furring strip will be unbroken)
You're going to start by coating the back of each furring strip with thickened epoxy resin and screwing the furring strips in place. Three things to mention here; always predrill the holes for the screws, always countersink the predrilled holes and start in the middle of the cabintop and work your way outboard.
When the first layer furring strips are glued and screwd to the underside of the cabin top, coat the back side of the remaining furring strips with glue and screw them into the first strips. The screws only touch the fiberglass cabintop and not create any new holes.
That was the hardest part.
Next, hop into your car or truck and go to Lowe's or Home Depot and purchase a 4x8 sheet of laminate (Lowe's has Formica, HD has Wilsonart, they're basically the same). If you have a truck big enough for it to lay flat, then perfect. If you're driving a little Ford Hiccup or a Toyota Burp, you might want to bring a razor knife with you to cut the sheet into manageable chunks. The sheet will roll up, but not very tightly. If it's too tightly rolled, it'll probably snap. To cut laminate, score the surfaced side TWICE, making sure you have cut deeply enough, then fold the two halves toward each other. It'll snap nice and clean.
Also buy a box of 1/2" stainless steel #6 oval-head screws and a box of #6 finish washers.
Cut the laminate (using the method above) to the dimensions you want (allow 1/4" gap between panels). When the panels are all cut, use some nicley finished teak battens screwed into the furring strips to hide the edges of the panels and finish them off with the screws and finish washers. (These will allow you future access to the panels without destroying the finish on the battens)
If you want to use vinyl instead, do everything exactly the same, but instead of buying laminate, get some thin (1/8" or 1/4") plywood instead and glue the vinyl to each panel. Pull the excess material around the edges and staple (stainless staples) it in place