Site was down overnight

[Phil Herring]

All of our sites were down from approximately 6pm Thursday to 3:30am Friday (PT).

The data center where our servers are located had a catastrophic power failure yesterday, and it must have been truly catastrophic. In addition to the abundance of power running into the building, there is literally a wall of batteries designed to keep all the servers running until a massive generator is started. For all of that to fail... well, I'm sure someone is looking for a job today.

Kudos to our heroic network administrator who was frantically working at 3am this morning to fix the mess caused by the servers shutting down improperly.

And of course, thanks for your patience.

UPDATE (for people who care about this stuff): We're being told the switch between the main power and the generator/UPS power exploded and wouldn't flow current either way.

 

[kpgraci]

That explains it. I just got a new Internet service this week and I tried to connect to SBO for the first time on my laptoplast night and I was shocked, shocked I say, to find the site unavailable...I was thinking all kind of bad things about AT&T, after all you have pretty good track record on uptime. Based on my experience, data centers have more outages than I ever expected - and explosions, of all things.

 

[Scott B]

I thought I maybe hit the wrong delete button......

 

[Steve Dion]

Maybe they need to test their fail-safes more often

 

[Rich Stidger]

I hate it when that happens....

 

[Phil Herring]

The irony comes from their constant claim of "no single point of power supply failure." Gentlemen, I think we have disproved that theory.

 

[Steve Dion]

Does anyone remember "The weakest link"? They may have found it "again".

 

[Deadhead]

Thought it was my electronically challenged mind . . .

Last night I got to the boat just after10p EDT. Over 85d in the boat. So I opened the thru hulls & started the AC. Figured I would sit in the cockpit & let things cool off down below while I checked the site. No SBO.com.

First I thought it was my iPad app. Then I thought it was my Droid app. So then I fired up the laptop & mifi. Still NFG! That's when I figured it wasn't on my end.

But I am still electronically challenged.

Glad to have SBO back up & running.

 

[JohnS]

Where I work, we've got a small server room with a UPS the size of a small SUV. Certainly sufficient power to allow time for clean shutdown of the servers if the utility power goes out.

One Saturday morning while at home, I started getting automatic text messages on my cell telling me the power got yanked out from under the servers, and when I tried to look into it, I couldn't even log into the UPS interface to figure out what happened. I called the guy in charge of maintaining the UPS and asked what the hell happened. He has no idea, his team wasn't working in there. Then I remembered that earlier on friday, the environment monitor in the room reported high humidity, and I had put in a call the facilities service team to come adjust the air handler in the room.

Apparently the "new" guy they sent out, noticed that he had to be admitted to the room with an electronic key card, and once inside and finished, thought he had to press the large red button near the inside of the door to exit - not realizing there was a motion sensor on the inside, and that the large red button (underneath the big clear plastic cover designed to prevent accidental use) was actually the EPO (Emergency Power Off) switch. It broke all the connections between the servers and UPS, servers and utility power, and UPS and utility power. Wham.

It took until the next Tuesday to check all the disks and get everything back up and running. Now we've got a big sign with 4" lettering labeling that switch.

 

[Scott B]

Actually it was an early April Fools joke and I banned everyone, including Phil

 

[Phil Herring]

(underneath the big clear plastic cover designed to prevent accidental use) was actually the EPO (Emergency Power Off) switch.

That's perfect. It's very hard to come with systems that will protect themselves from people.

 

[caguy]

Phil, speaking of systems was this forum out of a can or did you develop it from scratch? considering setting 1 up for our school psychologist at l a u s d.
When the site went down my wife applauded..

 

[Phil Herring]

This is a modestly modified version of vBulletin. I have mixed feelings about it. Very strong feature set, good moderation tools, lots of add-on modules available, but a bad interface in some critical areas (photo upload, for example) and a native search function that is completely worthless. However, the bad stuff may have been addressed in the latest upgrade.

 

[caguy]

Thanks,
Frank

 

[Crazy Dave Condon]

Phil;

When the series of bad storms about foureeks ago occured leaving parts of Ohio, MD (parts of Annapolis was without power for days), VA, NC, WVA, etc. I know the fellow who handles the repairs for Verizon's major trunk lines. There is a main center in northern VA which handles nearly all the 911 centers which took years to build. There were alot screaming to get that back on line which took days to rebuild what it took years to build. This is one reason I am staying with Verizon due to the dedication those guys to repair and get back on line quickly.

With record tempatures this summer, I wonder how many more storms we are going to experience. You will have a good laugh. The North Carolina state legislature in a sense has passed laws regulating mother nature from sea levels rising. Oh what a chuckle.

crazy dave condon

 

[Joe Diver]

It's always nice when an SPF identifies itself......rather than being found.....

 

[Chris Patterson]

Aw man, I had my virus scan searching, while crying the blues to beautiful bride about this "piece of junk" laptop, (it is not).
Side note--there's little telling what n.c. will do next. I'm defecting, claiming political asylum elsewhere..

 

[SEMIJim]

The last site I knew of that had (nearly?) the same thing happen at their hosting provider's was down for weeks, their database was so badly trashed.

Most of this forum software runs on a database management system that I, quite honestly, do not trust. IME it's simply not robust--esp. in the face of unexpected events of this nature. (I know there's nothing you guys can do about that. It is what it is.) But even with RDBMS software I more-or-less trust, I run a nightly database dump and keep X copies.

Jim

 

[Ken13559]

I was thinking, with1/2 of India suffers power failure, 670 million people affected, could it be SBO server is located in India? ...nnaah that cannot be. can it?