OVH - Fire destroys servers

Associate
Joined
13 Jun 2007
Posts
101
It was an unmanned DC, so I doubt it would be something like that.

As I said above OVH have been known to modify servers, watercool them, run them submerged, open case etc. I'd imagine either that, or an old battery somewhere swelling, exploding and inadequate fire suppression system combined with poor building design.

That's interesting, i didn't realise it was unmanned. Corners must certainly have been cut somewhere along the way which is surprising if they are modifying servers... you'd think you'd want all the safety system going to protect customers even though it is cloud based.
 
Soldato
Joined
29 Dec 2009
Posts
7,174
It's not an unmanned DC

Power cables started being replaced in various OVH DCs due to potentially faulty insulation so that's my bet.

Some of their DCs don't run active fire suppressant due to the way they operate cooling - i.e. not CRAC.

Migration of the cloud VMs located in SBG2 was being carried out but had to pull the plug to shutdown power.

Too many businesses have their disaster recovery managed by the same company that provides their primary server, I've no sympathy for those that have lost more than a day's worth of data.
 
Last edited:
Soldato
Joined
29 Dec 2009
Posts
7,174
Pretty sure I read in one article that it was, or at least very minimal manning.

There's a lot of hearsay going around but none of their DCs are unmanned. They do have fairly minimal on-site staff presence as standard though - in comparison to how many servers they maintain - but that's just the nature of their business, very production-line esque and minimal support provided, so it must work for them otherwise they wouldn't have the market presence and growth that they've managed.
 
Soldato
Joined
29 Dec 2002
Posts
7,175
At least nobody was seriously injured, though from the updates I read elsewhere it seems it’s affected more of the site than first reported. I like OVH, but this isn’t the first issue linked to its unorthodox approach to DC construction. Thankfully I rarely use them now, Germany only adds a few ms of latency and spec wise is better value, that is if Hetzner are willing to do business with you.
 
Man of Honour
Joined
15 Jan 2006
Posts
32,369
Location
Tosche Station
Are they using shipping container offices?

I know construction workers do it but that's construction workers not a tech company.

Datacenters can be/are built in blocks using iso containers. The idea is modularity, as far as I'm aware. Microsoft have definitely done this, although from a quick Google is seems they're moving away from the idea.
 
Permabanned
Joined
9 Aug 2008
Posts
35,707
Datacenters can be/are built in blocks using iso containers. The idea is modularity, as far as I'm aware. Microsoft have definitely done this, although from a quick Google is seems they're moving away from the idea.

Microsoft also played with servers under water in the sea if I remember correctly. There is no right and wrong way as long as they have no power loss and built in redundancy in case of a fire or some other disaster.
 
Man of Honour
Joined
15 Jan 2006
Posts
32,369
Location
Tosche Station
Microsoft also played with servers under water in the sea if I remember correctly. There is no right and wrong way as long as they have no power loss and built in redundancy in case of a fire or some other disaster.
Yeah they submerged one for a year two years, brought it out recently. I believe the test exceeded expectations, however there's something about pushing the heat directly in to the sea that makes me feel a bit uneasy, climate-wise. I know it's no different to it being pumped out in to the air, it's just different somehow in my head.
 
Permabanned
Joined
9 Aug 2008
Posts
35,707
Yeah they submerged one for a year two years, brought it out recently. I believe the test exceeded expectations, however there's something about pushing the heat directly in to the sea that makes me feel a bit uneasy, climate-wise. I know it's no different to it being pumped out in to the air, it's just different somehow in my head.

Can't be any more damaging than a fire in a DC lol
 
Soldato
Joined
5 Nov 2010
Posts
23,904
Location
Hertfordshire
Yeah they submerged one for a year two years, brought it out recently. I believe the test exceeded expectations, however there's something about pushing the heat directly in to the sea that makes me feel a bit uneasy, climate-wise. I know it's no different to it being pumped out in to the air, it's just different somehow in my head.

I’d be concerned for localised ecosystems that would be directly affected for sure.
 
Soldato
Joined
1 Apr 2014
Posts
18,532
Location
Aberdeen
Since this was in the news, I wonder if AMD have been on the phone? They could probably save a lot of money and restore service much more quickly by going with AMD's new Epyc chips. And if AMD gets some publicity, it's win-win.

Something along the lines of "OVH and AMD are pleased to announce that thanks to consolidation with AMD's new Epyc servers OVH's datacentre has been rebuilt and is back online ahead of schedule. Blah blah blah."

The fire got some coverage on Linus Tech Tips. Apparently Floatplane had servers there but despite some of their servers being destroyed they suffered no loss of service.


Main mention at 45:53
 
Back
Top Bottom