Interclickz Update

[woostar]

Well as you will have noticed Interclickz has been down for a few weeks now. This is a lot longer than we had planned.
I've kept Jon in the loop from day one, but to save anymore eye rolling I think you are all overdue for an update.

So where is Interclickz?

Bear with me because some of it gets a little technical in places (and in others it's down right bonkers):
On 24th May Interclickz' server went offline without any apparent reason. We tried the data centres website to access the support console and that was offline too. After numerous phone calls (all unanswered) and posts on webhosting sites we found out that the company was on the verge of going bankcrupt. After many more calls and posts we managed to contact the company that supplied the IP transit (which had been cut because of payment issues with the data centre resulting in Interclickz going offline) and they agreed to turn the connection back on for us (and all the others in the data centre) to reclaim as much as we could in the way of backups. We were warned that access to the servers would only last for 3 hours. Knowing that the chances of the server being sold off as liquidated stock we immediately backed up, downloaded and wiped the hard drive to safe guard against anyone else using our data...
This is where the problems started (keep reading, there are 3 parts):-
Half of compressed backup files were corrupt. Unfortunately in the rush to get everything off the server and wipe all the personal data, nobody realised this until after it was too late. All offsite backups were stored on a server owned by the same company (which were wiped too) which only left an our daily backups which were overwritten with corrupted backups on our nas drive and a few other maintenance backups that are on the office computer (keep reading there is light at the end of the tunnel).
OK, this was the first problem we had... so we took stock between all the backups (and what we could reclaim from the corrupt files) we started putting it all back together again. By this time we had bought a new server hosted in the US and were busy trying to produce a new working database... then we got hit by the zero day bug when Redhat Enterprise Linux (rhel5.2) caused our server to crash; taking us offline and we have to start reclaiming data again!! (keep reading the best is yet to come).
Part 3: After starting again from scratch and provisioning a new server we get less than 48 hours with a working server and the friggin' data center blows up! I swear it sounds like a poorly written movie but you can read about it here: Web Host Industry News | The Planet Explosion Causes Outages. This affected a lot of big names like Statcounter, B3ta and Hostgator but worst of of all put even further delays on getting Interclickz back up and running.

So what happens now?
Well the good news is we have a working server and a working file structure; now we are back to where we should have been weeks ago and recovering the database. I estimate that 70% of it has been repaired, including ALL referral and referrers (so don't panic you have lost no greens). We are now working the rest of the tables.

How much longer is this going to take?
The truthful answer to this is 'I have no idea' but hope not too much longer. The kind of recovery we are currently going through is a first even for me (and I've recoved others databases). With any luck I'll be able to give you another update soon.

Are my personal details safe?
You details are sooo safe even we are having trouble accessing them
All other servers were deleted and we are the only people that have access to them.

What could you have done to stop this happening?
I think you'll have to agree that this doesn't happen everyday, and is virtually impossible to plan for. If there is one thing we can learn from; it is to maybe look into the possibility of synchronising on another server (which has never been an issue because the server load has never warranted doing so).

None of this is our fault; are you planning on compensating your members for the downtime?
Once we have working database I'm sure we'll be able to come up with some ideas or promotions to help ease the pain.


Anyway, I just wanted to let you know what was going on and I'll be back soon to give updates as and when I have them.


Regards,
W.>

[Dim Tim]

Thanks Woo, the update is much appreciated

Looking forward to the competitions lol

[Jono310]

Thanks for the update, it shows us that you care!

hopefully it will all get back online soon so that you can have your business back, and we can get our gifts!!

all the best,

Jono

[Jon]

hen we got hit by the zero day bug when Redhat Enterprise Linux (rhel5.2) caused our server to crash;

To add, this bug in RHEL is a very very strange bug that never should have made it into a stable release of the software.

[mattflower]

Sounds like you've been stitched mate, negative replies aren't really needed. It's fairly obvious that this is causing you to lose money too

[tombsc]

The site's completely gone now

I hope you get things sorted soon woostar, this must be a real nightmare for you

[domel9]

Don't think its gone completely.. still says 'Maintenance'

[mattflower]

It's not gone, i'll still be supporting the site when it comes back online.
Everyone falls over, it's how you get up that counts

[Jono310]

Originally Posted by mattflower

It's not gone, i'll still be supporting the site when it comes back online.
Everyone falls over, it's how you get up that counts

too right.... well behind you Woostar, hope things are progressing for you!

J

[Jono310]

any news Woostar?

[Tippy]

Thank you for letting us know what was happening!

[chigley]

Completely missed this thread. Very bad luck mate, and hope the recovery isn't denting your accounts too much!

[-markie-]

whats the latest?

[Jon]

Unfortunately I keep missing woostar, so have not found the latest situation from him. However I do know he has been processing orders, make a note in Clickz Support - eXceem and I am sure he will make sure you get the current value of your greens if you prefer to just cash out.