PlusNet wipes 700gb of customer emails.
12 07 2006Quoted from Plus.Net:
This major incident occurred as a result of human error during work to resolve timeouts when collecting mails, which we reported via service status last week:
http://usertools.plus.net/status/archive/1152295291.htm
As of Sunday morning, things had progressed well, and we were on track to solve the issues with mail timeouts, which had started to occur following our move to a new email storage system.
At 8AM on Sunday morning our engineers were in a position to switch over to use of the new storage solution. As the first stage of this, an engineer was in the process of bringing the new back-up storage server into service. As part of the preparation of the mirrored disks on this platform the disks had to be reconfigured and all existing data on them removed.
At the time of making this change the engineer had two management console sessions open - one to the backup storage system and one to live storage. These both have the same interface, and until Friday it was impossible to open more than one connection to any part of the storage system at once. The patches we installed on Friday evening removed this limitation, but unaware of this, the engineer made an incorrect presumption that the window he was working in was the back-up rather than the live server. Subsequently the command to reconfigure the disk pack and remove all data therein was made to the wrong server.
Although this was noticed very quickly, over 700GB of live customer data was removed before the process could be halted.
I can’t begin to imagine what went through that poor technicians head when he realised he had wiped the wrong server. I often have several consoles open and run commands in both at the same time, but I always check which is which, especially when removing data!
Holy crap.
Poor guy. I did something like this once, I accidently formatted drive E instead of F. >_
rm -rf / doesnt help either, ive done that once