Downtime update

moviedrome · 07-06-09, 09:50 PM

At approx 7pm GMT VASERV HyperVM was hacked and it appears that all nodes have some level of damage. We are currently working on the situation and will be putting updates here. Currently we have no ETA on this

moviedrome · 07-06-09, 10:26 PM

From this update it appears we may not have lost the data after all

New update:
At approx 7pm GMT VASERV HyperVM was hacked and it appears that all nodes have some level of damage. We are currently working on the situation and will be putting updates here. Currently we have no ETA on this 23:18 GMT. We are going to bring the support desk back online shortly so we can start getting a track of where customers are. Per DC LA FSCKVPS - People are onsite working on the system WireSix Atlanta - People are working onsite TMS - Expecting someone onsite within 1 hour UK - We have 4 people onsite and gauging status Overall it looks like /boot on the nodes has been removed. Some nodes are definitly missing /vz data and others have it intact. We will be going node by node to get things going ASAP. Our HyperVM db's are intact so this means we can link everyone to their VPS

moviedrome · 07-06-09, 10:56 PM

latest:

23:56 GMT: We now have a rolling action plain in place for all nodes and are starting checks/restores. Please note we are expecting at least 24-48 hours to get things even remotly stable

moviedrome · 07-06-09, 11:36 PM

final update for tonight- i need my bed - sorry guys, i have to sleep too:

Basically - they might be able to get all the data back, maybe not - semi promising doesnt say a lot really does it?

00:32 GMT: We have so far done some test rebuilds on 5 boxes and results look semi promsing for the root VPS data (/vz). /etc/ was removed meaning config files need rebuilding however this is easy enough to do from HyperVM database. As it stands we will NOT be giving public access to HyperVM for the forseable future. We may/may not still use it internally via some very strong firewall controls. For rebuilds etc we will be asking people to do support tickets etc

Jon · 08-06-09, 05:57 AM

Just to update you a bit, FJ's hosting company have been bringing in people overnight to work on this server by server, I forgot to ask Rich what his server ID was. The latest update is

Quote:

05:22 GMT: We have approx 90% of the fsckvps nodes now online and are working on restroing VPS access. Currently we are putting everone on the same basic config just so things are up and will go round setting correct limits when things are calmer.
For LA vaserv ndoes all have been reloaded but need configs re-creating which we will do once our HyperVM VASERV is restored, through there won't be public access to start with, if at all. VAServ Texas are about all restored. UK we are about 50% of the way through and have 3 people working flat out on it

To add to that there is a big list of individual server's and their status, but I will leave it for Rich to post any specifics

moviedrome · 08-06-09, 07:05 AM

Unfortunately none of the servers listed are mine, i've finally managed to get a support ticket in but i imagine that they are inundated so I don't expect a quick response

moviedrome · 08-06-09, 08:03 AM

8:50am they've started on the VZ UK servers (which is where mine is located)

moviedrome · 08-06-09, 09:28 AM

10:20 - they've restored servers 1-10 in the UK (i'm not sure where i sit as my server is dedicated and has a name not a number) but everyone is on the case and you can rest assured we are doing everything in our power to get FJ back up with no data loss.

moviedrome · 08-06-09, 04:58 PM

Sorry for the lack of updates, they are v busy trying to get the whole network back up and they just give stock replies at the moment. As soon as I hear anything new I will post here, you have my assurance I'm trying my best.

moviedrome · 09-06-09, 09:28 AM

no more news I'm afraid other than to put it in context. The creator of hypervm (the software which allowed the hack) has hanged himself. That's how serious this is.

moviedrome · 09-06-09, 11:07 AM

Latest: it appears out server has been compromised they are reinstalling the OS and trying to recover the data at the moment.

moviedrome · 09-06-09, 05:44 PM

I know how frustrating it is but the situation is still that the server is officially up so not damaged as much as the others with full data loss. Our hosts are trying to recover all data (so none lost which is obviously the best solution). The hosts are also trying to getme access to the 24 hour old backup which is second best. I don't want to revert to the 72 hour old backup unless there is no other solution. If no change by tomorrow afternoon this is what I'll have to do though.

moviedrome · 10-06-09, 01:03 PM

The latest: They believe our server to have suffered 100% data loss and it cannot be recovered. The backups are present apparently but I still can't get access - i've been told to try again in 15 minutes. Literally as soon as I have the backups I can start the rebuild.

Please accept my wholehearted apologies, I am devastated that this has happened and I'm just about at the end of my tether with the hosts. Jon has kindly offered me temporary exceem hosting and has set me up with an account so I can crack straight on with it so thanks to him for that. Thanks also to Lee (Kudos) for his kind offer of help at this difficult time for us as well.

moviedrome · 10-06-09, 03:49 PM

Finally some good news! Our hosts have managed to retrieve the backup that is only 11 hours old. They are in the process of uploading it for me now (but it's big - 516mb zipped up) - i'll then have to download it, make sure it's the right one and reupload unzipped and then we're on the road to recovery - i've already updated our nameservers so some of you might start to get a holding page instead of the 404 error but it's looking much more positive now

moviedrome · 11-06-09, 04:11 PM

and we're back!

Tortuous process, and certain things may not work 100% correctly but i'm relying on you guys to point them out to me

Unfortunately we have lost 11 hours worth of data but at one stage it looked like it might be 72 hours data and a site version from october 08 so all things considered it's not too bad.

I can only apologise for the amount of downtime and the data loss, was totally out of our hands and everyone has worked really hard to get it back up as soon as possible. I believe we did the right thing waiting for the later backup as it minimised the loss to the customer.

A massive thanks go to Jon for letting us use exceem's server until we find a permanent home and for helping me out with all the technical issues in doing that and for listening to me rant.

Another big thanks goes to Charlie for putting up with my constant texting when he should have been focusing on his exams and the offer of hosting.

And again a big thanks to Lee from Kudos as well for the offer of hosting.

It's been a very difficult few days, and thank you all for your patience and best wishes.

moviedrome · 11-06-09, 04:13 PM

I've reopened this thread btw for questions.

Shawikebab · 11-06-09, 04:14 PM

Its not workin :S
I cant get onto the site, i get 404

TheDetox · 11-06-09, 04:14 PM

status banners back for gadgets and gifts

Mak · 11-06-09, 04:15 PM

I'm on

Just got into gifts

For me....

Gifts up
gadgets up
apple up

consoles down

moviedrome · 11-06-09, 04:15 PM

Quote:

Originally Posted by Shawikebab

Its not workin :S

in what way? Sorry i should have said - people will need to give me specifics of the problems

07-06-09, 09:50 PM	#1
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	Downtime update At approx 7pm GMT VASERV HyperVM was hacked and it appears that all nodes have some level of damage. We are currently working on the situation and will be putting updates here. Currently we have no ETA on this __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

07-06-09, 10:26 PM	#2
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	From this update it appears we may not have lost the data after all New update: At approx 7pm GMT VASERV HyperVM was hacked and it appears that all nodes have some level of damage. We are currently working on the situation and will be putting updates here. Currently we have no ETA on this 23:18 GMT. We are going to bring the support desk back online shortly so we can start getting a track of where customers are. Per DC LA FSCKVPS - People are onsite working on the system WireSix Atlanta - People are working onsite TMS - Expecting someone onsite within 1 hour UK - We have 4 people onsite and gauging status Overall it looks like /boot on the nodes has been removed. Some nodes are definitly missing /vz data and others have it intact. We will be going node by node to get things going ASAP. Our HyperVM db's are intact so this means we can link everyone to their VPS __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

07-06-09, 10:56 PM	#3
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	latest: 23:56 GMT: We now have a rolling action plain in place for all nodes and are starting checks/restores. Please note we are expecting at least 24-48 hours to get things even remotly stable __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

07-06-09, 11:36 PM	#4
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	final update for tonight- i need my bed - sorry guys, i have to sleep too: Basically - they might be able to get all the data back, maybe not - semi promising doesnt say a lot really does it? 00:32 GMT: We have so far done some test rebuilds on 5 boxes and results look semi promsing for the root VPS data (/vz). /etc/ was removed meaning config files need rebuilding however this is easy enough to do from HyperVM database. As it stands we will NOT be giving public access to HyperVM for the forseable future. We may/may not still use it internally via some very strong firewall controls. For rebuilds etc we will be asking people to do support tickets etc __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

08-06-09, 07:05 AM	#6
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	Unfortunately none of the servers listed are mine, i've finally managed to get a support ticket in but i imagine that they are inundated so I don't expect a quick response __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

08-06-09, 08:03 AM	#7
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	8:50am they've started on the VZ UK servers (which is where mine is located) __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

08-06-09, 09:28 AM	#8
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	10:20 - they've restored servers 1-10 in the UK (i'm not sure where i sit as my server is dedicated and has a name not a number) but everyone is on the case and you can rest assured we are doing everything in our power to get FJ back up with no data loss. __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

08-06-09, 04:58 PM	#9
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	Sorry for the lack of updates, they are v busy trying to get the whole network back up and they just give stock replies at the moment. As soon as I hear anything new I will post here, you have my assurance I'm trying my best. __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

09-06-09, 09:28 AM	#10
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	no more news I'm afraid other than to put it in context. The creator of hypervm (the software which allowed the hack) has hanged himself. That's how serious this is. __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

09-06-09, 11:07 AM	#11
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	Latest: it appears out server has been compromised they are reinstalling the OS and trying to recover the data at the moment. __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

09-06-09, 05:44 PM	#12
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	I know how frustrating it is but the situation is still that the server is officially up so not damaged as much as the others with full data loss. Our hosts are trying to recover all data (so none lost which is obviously the best solution). The hosts are also trying to getme access to the 24 hour old backup which is second best. I don't want to revert to the 72 hour old backup unless there is no other solution. If no change by tomorrow afternoon this is what I'll have to do though. __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

10-06-09, 01:03 PM	#13
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	The latest: They believe our server to have suffered 100% data loss and it cannot be recovered. The backups are present apparently but I still can't get access - i've been told to try again in 15 minutes. Literally as soon as I have the backups I can start the rebuild. Please accept my wholehearted apologies, I am devastated that this has happened and I'm just about at the end of my tether with the hosts. Jon has kindly offered me temporary exceem hosting and has set me up with an account so I can crack straight on with it so thanks to him for that. Thanks also to Lee (Kudos) for his kind offer of help at this difficult time for us as well. __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

10-06-09, 03:49 PM	#14
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	Finally some good news! Our hosts have managed to retrieve the backup that is only 11 hours old. They are in the process of uploading it for me now (but it's big - 516mb zipped up) - i'll then have to download it, make sure it's the right one and reupload unzipped and then we're on the road to recovery - i've already updated our nameservers so some of you might start to get a holding page instead of the 404 error but it's looking much more positive now __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

11-06-09, 04:11 PM	#15
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	and we're back! Tortuous process, and certain things may not work 100% correctly but i'm relying on you guys to point them out to me Unfortunately we have lost 11 hours worth of data but at one stage it looked like it might be 72 hours data and a site version from october 08 so all things considered it's not too bad. I can only apologise for the amount of downtime and the data loss, was totally out of our hands and everyone has worked really hard to get it back up as soon as possible. I believe we did the right thing waiting for the later backup as it minimised the loss to the customer. A massive thanks go to Jon for letting us use exceem's server until we find a permanent home and for helping me out with all the technical issues in doing that and for listening to me rant. Another big thanks goes to Charlie for putting up with my constant texting when he should have been focusing on his exams and the offer of hosting. And again a big thanks to Lee from Kudos as well for the offer of hosting. It's been a very difficult few days, and thank you all for your patience and best wishes. __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

11-06-09, 04:13 PM	#16
moviedrome FJ Network Owner Join Date: Jan 2007 Posts: 5,236 Thanks: 1,307 Thanked 5,669 Times in 1,798 Posts	I've reopened this thread btw for questions. __________________ IT'S SHOULD HAVE NOT SHOULD OF! Consoles \| Apple \| HDTV \| PS3 \| Mobiles \| Sports \| Gadgets \| Points \| Gifts \| Select Supported Network of the Year 2008 & 2009 Network Staff Member of the Year 2009 Supported Network of the Year (Design) 2008 New and Unproven Network of the Year 2007
Offline

11-06-09, 04:14 PM	#17
Shawikebab iPod 20gb Join Date: May 2009 Location: Anfield - KOP end Posts: 422 Thanks: 51 Thanked 54 Times in 36 Posts	Its not workin :S I cant get onto the site, i get 404
Offline

11-06-09, 04:14 PM	#18
TheDetox The Method! Join Date: Nov 2006 Location: Kent Posts: 6,019 Thanks: 1,416 Thanked 774 Times in 678 Posts	status banners back for gadgets and gifts __________________
Offline

11-06-09, 04:15 PM	#19
Mak It's A Free World ! Join Date: May 2006 Location: 3rd Rock From The Sun Posts: 2,125 Thanks: 563 Thanked 178 Times in 146 Posts	I'm on Just got into gifts For me.... Gifts up gadgets up apple up consoles down __________________ Spoiler Total = Approx £5,000 www.free-console.co.uk Last edited by Mak; 11-06-09 at 04:19 PM..
Offline