Network Outage

[grant]

Hi Dave and Roger

I understand what you are saying!

I was worried that with the back-up being to the end of April we could end up posting and those new posts splitting the back up and the later restored posts.

It will be a traumatic few days for HWI'ers - not being able to post - but the logic is clear, once everything is working properly we will have the forum in its correct order!

You guys are doing a magnificent job and you know without saying the level of our appreciation is beyond words. :applaud: :applaud: :applaud:

As Mark has said, anything we can do to help.

Best wishes
Grant

[admin]

Thanks for your offer of help, guys. Until the times comes, we won't know exactly what is needed and we may have to play it by ear, as it were. Time will tell.

Apparently ALL ProBoards servers are now back online. We are merely awaiting the data to be transfered to restore everything up to the end of April. Hopefully, that might happen some time tomorrow.

It is ironic that I have spent so much time today doing some sort of forum-related work when one of our forums is offline! And considering I only got two hours' sleep last night, I think I'll... I'll... [yawn]... :snoring:

G'night all!
Roger

[Joe]

Jun 1, 2008 21:04:11 GMT admin said:

Northern Lights, you're closer to Houston that I am. Go and sort them out, would you?

You know, when I saw the data center was located in Houston, Texas...that's where the Johnson Space Center is located! And the Space Shuttle Discovery was launched from Florida about an hour before the server outage. JSC controls the shuttle from Houston too...quite a busy place.

We always blamed the launches on making it rain but this time...

I also want to say that it was nice for most of the HWI forum staff and several prominent members to re-convene at the AAI forum. It was as if our physical meeting place had a small fire...well, we'd meet up across the street for example.

I concur with this plan of action that Roger and Dave have outlined. I shall continue to watch for further developments.

Joe

[Northern_Lights]

Jun 1, 2008 21:04:11 GMT admin said:

Northern Lights, you're closer to Houston that I am. Go and sort them out, would you?

Sorry Roger, I've got school tomorrow! Not that I'd be able to do it anyway....

Still not up. Poor HWI. I miss it.

[stephany]

Hi everyone,

It's a bit weird to wake up without HWI But I just got an email from CD Japan to let me know that they sent my new Hayley CD so it compensated a little bit ;D I'm impressed by their service and I look forward to receiving the album.

Thank you very much Roger & Dave for all the latest updates!

Stephany

[admin]

Good morning everyone,

Here is a brief extract from a statement made by Douglas J. Erwin, Chairman & Chief Executive Officer of The Planet in Houston made at 11:14pm (CDT) yesterday (7:14am BST today):

While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed.

There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running.

We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.

I am now checking for further updates and will report back regularly.

Roger

[richnorri]

Good morning Roger and everybody!

I've just caught up with this thread, and it all makes good sense to me. Hopefully HWI will be back to normal soon, meanwhile it looks like another early breakfast.

I still can't access the HWI Forum at the moment, but I'll be available to assist as necessary when I can.

Richard

[admin]

Thank you Richard. Enjoy your breakfast!

Here is a summary of the situation as I understand it:

1. ProBoards have installed 24 new servers and they are all online. This has enabled them to display the message which you can see if you try to access the HWI Forum.

2. ProBoards are currently transferring data from their back-up to those new servers. When that is done and it has propagated around the world, the forum will be restored up to the end of April. PLEASE DO NOT POST UNTIL YOU GET THE GO-AHEAD FROM EITHER ME OR DAVE.

3. As soon as the forum appears on my screen, I will temporarily disable posting to minimise any long-term consequences. The forum will then be available to view only. From then on, any updates will appear on a special board that I will create on the HWI forum and members WILL be able to post to that one thread (even though posting will be disabled to the rest of the forum).

4. ProBoards will transfer data from the old servers to the new ones. This will restore everything in full up to the time of the explosion. However, this transfer cannot begin until power is restored to the old servers and, as we have heard this morning, there is severe damage to the cabling.

5. As soon as the final data transfer has been completed, I will enable posting and we will be fully operational.

Roger

[grant]

Good morning everyone!

Thank you Roger for the latest update - your plan of action seems to me to be the only logical one or we could end up with posts and threads all over the place.

Personally, I feel OK at the moment, like Joe said, we've got somewhere to be even if it aint where we want to be, but we can communicate and that's the main thing.

It feels like we're waiting for a close friend or relative to have major surgery and are keen to know the outcome. At least we know that everything that can be done is being done!

Stephany, I also got the Email from CD Japan this morning :applaud: :applaud: Their shipping service is second to none and usually very quick so I'm looking forward to prompt receipt.

I see also Stephany that, despite the current adversity, you are finding some Hayley pics for us - bless you!

Anyway guys and gals, I have somewhere important to be in a few hours

See y'all
Grant :exit:

[martin]

I would just like to echo everyone's thanks to The Planet staff for their dedication during these unprecedented times. I never realised the scale of the operation (6000 servers) and it proves that thorough disaster recovery procedures were not only in place but also proven to be working.

Also thanks to Roger and Dave for the frequent updates and their own sensible management plan to support our own site recovery. Roger now needs a personal recovery plan for his lost sleep!

Martin

[admin]

I have found a couple of forums that use Server 1 which are now fully in view (but only up to the end of April). One of them displayed a request that members refrain from posting until further notice so we are not alone in taking this precaution!

Forums using Server 2 still only display the message from ProBoards. I haven't looked beyond that but will keep monitoring the situation as it may give us a clue as to when Dave or I have to spring into action! Some hopes, yeah, I know!

Roger

[comet]

Ha ! Hi Folks

found you all again,
all playing "internet hide and seek" ?

Best wishes from Dublin Ireland

[admin]

Here is a brief update posted by Martyn Dale of ProBoards a few minutes ago:

The first group of forums will start to be brought online within the next 30 minutes.

Unfortunately some servers are in part of the Data Center which had more damage to the sourounding circuitry than other parts.

The Planet at this point expect that part of the DC to be back online within 18 hours

What isn't clear is which part of the DC our server is in. Nevertheless, indications are that data transfer will commence soon and we could be online within 18 hours from now.

Roger

[admin]

If I have understood the latest update correctly, the news is not good. It seems that our server is in the area of the Data Centre that is worst affected. Apparently, that puts us in the category they are calling "H1 Phase 1". (H1 is Data Centre No 1 in Houston as opposed to Dallas). They now believe that power will be restored to that evening tomorrow evening, but that is CDT so we are talking of Wednesday morning UK time. And that is merely for the data transfer to begin.

Sorry folks, but it looks as though we may have a long wait yet.

Server Restoration Progress

Roger

[Dave]

Hi Roger and everyone,

I think it could all be completed on Tuesday - so we'll find out tomorrow which guess is right

Cheers

Dave :jester: