Complete DR Solution

[Spammeh]

So I've been tasked with completely revamping our DR system at work and if I'm honest its a bit of a mess. Currently have the following at my disposal

HEAD OFFICE
Backup Exec 11d...
LTO3 tape drive
LTO4 tape drive

COLOCATION
2x ESXi 5.0 hosts with 32GB RAM each
QNAP TS-1279U-RP (10GB RAM after RAID5)
Connected to head office via 10MB IPVPN

What I'm trying to achieve is a 3-pronged backup / high availability solution with the following

• Disk-2-Disk Array for quick restore
• Large LTO tape autoloader
• high availability across the IPVPN link

Correct me if I'm wrong, but this will allow me to have long term snapshots of the data (using tape), quick restore snapshots (via disk array) and high availability across the 10MB IPVPN at our Colo in case of major failure.

Hope all the above makes sense up to now...

The budget is £40k for this project, and as far as I can see I need to buy the following:

• LTO autoloader
• Rackmount backup server
• Disk array with masses of cheap storage
• Backup software (vsphere and physical) that also provides a high-availability option
• 2x replacement ESXi hosts for the Colocation

A majority of our data is on virtual servers (sitting on an IBM v7000 SAN), however we do have a number of physical servers, and servers hosted at remote sites.

------

As you can imagine, this could be quite a challenge, as we have (in an ideal world) around 10TB of data to backup. As it stands, most things aren't backed up that should be - and that makes me worry.

Question is, does anyone else run a similar solution at their place? What disk arrays are you using? What software do you use?

I'm not after a definitive answer, or for you guys to do my job, but its good to get like-minded individual's opinions on what they have used, etc.

 

[Deleted member 138126]

Not sure if I fully understood the situation, but you're going to struggle with 10 Mbps (an absolute maximum of 108 GB/day). What are you copying/replicating to the remote site?

http://web.forret.com/tools/bandwidth.asp?speed=10&unit=Mbps

 

[Oh Ed not again.]

Are you planning on using Vmotion or anything like that for your ESX hosts?

 

[Spammeh]

Not sure if I fully understood the situation, but you're going to struggle with 10 Mbps (an absolute maximum of 108 GB/day). What are you copying/replicating to the remote site?

http://web.forret.com/tools/bandwidth.asp?speed=10&unit=Mbps

We're not going to replicating all of the 10TB over that line to the Colocation. 108GB a day would be plenty, as there isn't 108GB's worth of change on a daily basis.

 

[Spammeh]

Are you planning on using Vmotion or anything like that for your ESX hosts?

We do have vmotion across a majority of our virtual environment

 

[Hellsmk2]

I'm not going to get into the logistics of your plan (I wouldn't personally do it the way you've described; backup and DR are two separate concepts), but I will say look at VMware Site Recovery Manager.

http://www.vmware.com/uk/products/datacenter-virtualization/site-recovery-manager/overview.html

If you want to do in-house DR, you won't find a better solution. It is also fairly cheap, so well within your budget. Vsphere integration means it is also simple to set up and maintain.

 

[aln]

It's obvious but whatever you do, your DR plan should include auditing the thing actually works on a regular basis.

 

[Spammeh]

Yes, we'll try and do a failover test on a quarterly basis I think. Why wouldnt you mix backup and DR? A lot of the big backup software offers them together, and surely that is logical, as they use similar technologies to get the end product?

 

[Deleted member 138126]

We're not going to replicating all of the 10TB over that line to the Colocation. 108GB a day would be plenty, as there isn't 108GB's worth of change on a daily basis.

The problem isn't replicating the amount of daily change. The problem is when you need to re-sync from scratch, which can happen with any replication technology for a vast number of reasons. It would take 92 days under absolute ideal conditions to replicate 10 TB across a 10 Mbps link -- in other words, as good as not having any sort of replication at all.

Don't underestimate this problem.

A few random things to think about: How will you be doing the initial sync? What happens if someone dumps a large amount of data on to replicated storage? What happens when the business asks for a handful of VMs to be built (15 - 20 GB per VM)?

108 GB is 1% of 10 TB -- do you think it is realistic that only 1% of your data will change every day? Also, 108 GB is spread across 24 hours, but in reality, the majority of changes will probably occur between 7 and 7, unless you have some sort of heavy overnight batch process. This immediately doubles your bandwidth requirements (to transfer 1% of 10 TB during 12 hours, you need a 20 Mbps link at full throttle).

All these numbers are theoretical maximums, and in reality you probably get 70-80% of the link at best.

Here are a couple of links discussing VM replication and bandwidth requirements:

http://blogs.vmware.com/vsphere/2011/10/vsphere-replication-bandwidth-1.html
http://blogs.vmware.com/vsphere/2011/10/vsphere-replication-bandwidth.html

 

[Deleted member 138126]

Why wouldnt you mix backup and DR? A lot of the big backup software offers them together, and surely that is logical, as they use similar technologies to get the end product?

DR and Backups are completely different things.

DR is more like RAID, i.e. it protects you against disastrous failure (hardware/electricity/earthquake) -- hence the word "disaster". This will not protect you if someone accidentally deletes a file, or saves on top of a file, or a file gets corrupted; especially if you only just noticed but it happened six weeks ago. That is what Backups are for.

 

[richard smith]

Look at veeam backup and replication. Obviously this works only for virtual stuff and I think you'll struggle with that small a pipe.

You need (as a business) to review the risk, define critical applications and servers, define the maximum down time you can live with and use that as a starting point for working out how much data you're talking about needing to transfer.

Say you were able to get everything up and running in the colo.. whats the traffic profile of the data your users would need to access down a 10 meg pipe? CIFS, SQl, Exchange etc..

I assume you've got file servers as a minimum. Can you imagine what it'll be like for a large bunch of users accessing all their data over a 10 meg pipe?

 

[Spammeh]

Guys, I won't be backing up 10TB over to our remote site. The systems that will be replicated are far smaller than 10TB and have been identified by the business as business critical. Yes, the initial replication will be painful, however that could be mitigated by taking the initial backups physically over to the Colocation. The total amount of data to be replicated to our Colocation would be approx 3TB, with the daily data change being between 20 and 35GB.

@rotor, I know DR and backups are totally different methodologies, one is a snapshot at a given time, and DR (close to realtime) requires a live mirror of current data. However, as far as looking at technologies and software that can provide these for the business are often bundled together and can utilise similar technologies to achieve this.

@Richard_Smith I've looked at Veeam and yes it would allow me to a majority of what I need to do, but it deals with Virtual only, and i have physical servers which are too precious to simple ignore when it comes to this.

 

[Hellsmk2]

Look at veeam backup and replication. Obviously this works only for virtual stuff and I think you'll struggle with that small a pipe.

You need (as a business) to review the risk, define critical applications and servers, define the maximum down time you can live with and use that as a starting point for working out how much data you're talking about needing to transfer.

Say you were able to get everything up and running in the colo.. whats the traffic profile of the data your users would need to access down a 10 meg pipe? CIFS, SQl, Exchange etc..

I assume you've got file servers as a minimum. Can you imagine what it'll be like for a large bunch of users accessing all their data over a 10 meg pipe?

Veeam over a 10mb line will be useless. Only consider Veeam if you have a minumum of 100mb, and to really see it working, you'd really need a 1gb pipe.

It is a cracking product if you have the bandwidth, but utterly useless if you don't

 

[Yamahahahahaha]

Veeam over a 10mb line will be useless. Only consider Veeam if you have a minumum of 100mb, and to really see it working, you'd really need a 1gb pipe.

It is a cracking product if you have the bandwidth, but utterly useless if you don't.

I'd recommend Exagrid with Veeam in those situations.

 

[danb21t]

NetApp with Snapmirror
Keeping Backup Exec

 

[Spammeh]

Ive been looking at Unitrends recently. Looks a fantastic little product - seems to cover off most things from a software perspective - anyone used it?

 

[gavind]

I checked on them once but haven't tried yet. They boasted a 99% customer satisfaction rate which is a good one for starters.

 

[tbz_ck]

Ive been looking at Unitrends recently. Looks a fantastic little product - seems to cover off most things from a software perspective - anyone used it?

Yes, both their appliances and their software is excellent and very highly regarded in the marketplace. Check them out on the Spiceworks community for example.

 

[Tarqy]

NetWorker and DataDomain could be an option

 

[DRZ]

Veeam over a 10mb line will be useless. Only consider Veeam if you have a minumum of 100mb, and to really see it working, you'd really need a 1gb pipe.

It is a cracking product if you have the bandwidth, but utterly useless if you don't

Rubbish.

I have Veeam backups running happily on way less than 100mbit from some of my locations. Seed the backup first and you're fine as the compression and deduplication is fantastic. We also have WAN accelerators which do help but that's just belt and braces really.


I can't help but think that people are missing the point a bit here. Reversing the sync in the event of a smoking crater disaster isn't your P1 in such a situation. Restoration of critical services *is*. In the event of a total disaster, he should be in a position to flip the switch and have the production workload move to the DR site.

At that point, the question is can that 10mbit pipe cope with the workload? Depends on the workload but I know with proper QoS and so on I could get my critical services going over 10mbit (but it wouldn't be pretty) and from the sounds of it my environment is 100x the size of what is being discussed here.

That buys him time to get his primary back operational. That could be hours or days or even longer depending on what has happened. When the primary comes back, he restores from backup tape (thus seeding the reverse replication...) and starts to sync. At some convenient time he flips back to the primary site and service is restored.

I have never used it but I've looked at the PlateSpin Forge product before and its a pretty neat solution for protecting virtual and physical workloads. Supposedly not that expensive either...