Questions tagged [disaster-recovery]
Disaster recovery and preparedness is an unfortunate aspect of systems administration. This tag should be used for help with planning, implementation and best-practices related to recovering from a catastrophic event on a server or in a datacenter environment.
359
questions
150
votes
9
answers
167k
views
Monday morning mistake: sudo rm -rf --no-preserve-root /
Please note: The answers and comments to this question contains content from another, similar question that has received a lot of attention from outside media but turned out to be hoax question in ...
- 1,613
122
votes
13
answers
7k
views
Engineers are using explosives to remove hard rock outside our office building. What countermeasures should we take?
Our building is located approx. 100 meters from the explosive charges. They happen several times per day, and really shake the entire building a lot. This is going to go on for many days and the ...
- 1,553
51
votes
6
answers
26k
views
How to backup GPG?
What are the critical files I need to backup from GPG? I guess my private key would qualify of course, but what else?
- 1,849
40
votes
20
answers
9k
views
What's your checklist for when everything blows up?
Users can't get to their e-mail, the CEO can't get to the company's home page, and your pager just went off with a "911" code. What do you do when everything blows up?
- 1,506
36
votes
10
answers
69k
views
Unmount a nfs mount where the nfs server has disappeared
Server A used to be a NFS server. Server B was mounting an export of that. Everything was fine. Then A died. Just switched off. Gone. Vanished.
However that folder is still mounted on B. I obviously ...
- 31.9k
35
votes
7
answers
2k
views
What things do you look for when picking a server hosting company?
We are going through an RFP process of changing hosting companies for most of our servers (~10 fairly powerful workhorses and database servers).
When the existing company was picked I wasn't at the ...
32
votes
4
answers
3k
views
My server room has flooded
We recently went through a hurricane and our server room became flooded. Hooray for insurance. Anyway, I need to save as much data off one of the hard drives as possible. Yes, it was submerged for ...
- 445
29
votes
11
answers
2k
views
Disaster recovery plan development best practicies or resources? [closed]
I have been tasked with leading a project regarding updating a old and somewhat onesided disaster recovery plan. For now we're just looking at getting the IT side of DR sorted out. The last time ...
- 2,845
26
votes
5
answers
10k
views
BBWC: in theory a good idea but has one ever saved your data?
I'm familiar with what a BBWC (Battery-backed write cache) is intended to do - and previously used them in my servers even with good UPS. There are obvously failures it does not provide protection for....
- 21.9k
26
votes
2
answers
2k
views
Retrieving an RSA key from a running instance of Apache?
I created an RSA keypair for an SSL certificate and stored the private key in /etc/ssl/private/server.key. Unfortunately this was the only copy of the private key that I had.
Then I accidentally ...
- 2,725
19
votes
9
answers
6k
views
Architecture for highly available MySQL with automatic failover in physically diverse locations
I have been researching high availability (HA) solutions for MySQL between data centers.
For servers located in the same physical environment, I have preferred dual master with heartbeat (floating ...
- 23.8k
17
votes
9
answers
1k
views
Documentation As-A-Manual vs. Documentation As-A-Checklist
I've had discussions in the past with other people in my department about documentation, specifically, level-of-detail and requirements. In their view, documentation is a simple checklist of Y things ...
- 14.6k
15
votes
6
answers
29k
views
How to recover from a drive failure in a RAID 5 configuration?
This morning a drive failed on our database server. The drive array (3 disks) is setup in a RAID 5 configuration.
While we wait for a drive replacement we are preparing for a recovery strategy. Users ...
- 537
15
votes
7
answers
3k
views
Setting up a new backup scheme
I'm in the process of designing my first ever backup scheme. I'm completely new to managing data backup, and there are some concepts that I don't totally understand. Here's what I've got so far, and ...
- 221
14
votes
4
answers
804
views
IT lead does not have a backup, DR plan in writing [closed]
This is a general management question to IT managers out there.
We are a small firm with about 4 servers in our colo cabinent. No full time IT manager. But we do have one person on monthly contract ...
- 259
13
votes
3
answers
15k
views
Battery Backed Write Cache
I recently got some U server price quotes and some of them include BBWC:
What exactly does it do?
Is it just for RAID configurations?
If there is a power malfunction, isn't the data loss inevitable?...
- 1,226
12
votes
3
answers
38k
views
How to actually use mysql slave as soon the master is failover or got burnt
I have MySQL master-slave replication that works fine; I googled the whole net and MySQL site to find the standard procedure to make use of the replication but found nothing. It is as if admins are ...
- 254
12
votes
4
answers
5k
views
How do I backup my TRAC installations?
We use separate TRAC instances as our ticket system for many projects and need to have them moved off site several times a day for disaster recovery.
What is the best way to make this happen? Is ...
- 241
11
votes
12
answers
688
views
What's the first thing you check when an untouched unix server starts going berserk?
So you have this neatly setup unix server and it's super fast and works swell and everything is great for months, and suddenly all kinds of weird errors start showing up for a variety of different ...
- 4,632
11
votes
5
answers
673
views
High server availabilty for a small business
After having a bit of scare with a server that wouldn't come up one morning, the higher ups have decided that the business needs a high availability / fail over setup.
We have 5 main servers (4x ...
- 175
9
votes
3
answers
3k
views
Database accidentally deleted with a bash script [duplicate]
Edit: a follow-up question: Restore mongoDB by --repair and WiredTiger.
My developer committed a huge mistake and we cannot find our Mongo database anywhere in the server.
He logged into the server, ...
- 337
9
votes
3
answers
9k
views
Active Directory disaster recovery with DPM
I have a sort of catch-22 question here.
Suppose I'm using Microsoft System Center Data Protection Manager (2010 or 2012, it works the same way) to backup, amongst various other things, my Active ...
- 70.7k
8
votes
1
answer
2k
views
Does one failed drive + one single bad sector destroy an entire RAID 5?
During planning my RAID setup on a Synology Disk Station I've done a lot of reading about various RAID types, being this a great reading: RAID levels and the importance of URE (Unrecoverable Read ...
- 295
8
votes
1
answer
4k
views
Recover data from SCSI hard disk
We've got an old server with SCSI hard disk. The server crashed last week and it isn't exactly known what hardware component is damaged. Since the server is due to be retired anyway we don't want to ...
- 101
8
votes
3
answers
2k
views
Backing up VirtualBox VMs
Does anyone have a good complete strategy for backing up a bunch of virtual machines running under VirtualBox?
I intend to run a handful of virtual machines on a single hardware platform and back ...
- 905
8
votes
1
answer
4k
views
Recovery strategy for Master-Master replication
I have implemented a HA solution for mysql based on master-master replication. There is a mechanism on the front end part which guarantees that only one db will be read/written to at a given time (i.e....
- 243
7
votes
3
answers
7k
views
If DNS Failover is not recommended, what is?
As a followup question to his very popular question: Why is DNS failover not recommended?, I think it was agreed that DNS failover is not 100% reliable due to caching.
However the highest voted ...
- 511
7
votes
2
answers
4k
views
How to recover data from an Exchange 2013 database after a complete Active Directory loss?
Scenario: a single Exchange 2013 server in a Windows Server 2003 AD domain; one DC malfunctioned months ago and was dismissed (without proper demotion, no less); the other DC died yesterday and there ...
- 70.7k
7
votes
1
answer
3k
views
How do I configure a stretch cluster without shared storage between two sites?
I am trying to redesign our IT infrastructure and seeking help in implementing DR solution for our company.
I see that as 2 data centers in active-passive mode with the data replication. Currently ...
- 155
7
votes
5
answers
3k
views
Local to Remote Webserver Failover
Short and sweet, I don't suppose you'll need more detail than this:
We host our website on an in-house
webserver.
A catastrophe has and will happen
again where communication from the
web into/out ...
7
votes
2
answers
6k
views
Hadoop HDFS Backup & DR Strategy
We are preparing to implement our first Hadoop cluster. As such we are starting out small with a four node setup. (1 master node, and 3 worker nodes) Each node will have 6TB of storage. (6 x 1TB disks)...
- 221
7
votes
2
answers
2k
views
In 2020 - are there any viable Linux block-level replication alternatives for DRBD? [closed]
I'm researching how can we implement near-realtime replication from primary datacenter to a disaster recovery site. Data that would get replicated would be:
Images of KVM VMs
MySQL and PostgreSQL ...
- 30.1k
6
votes
5
answers
7k
views
What's the difference between a Disaster Recovery Plan and a Business Continuity Plan?
I used to think both terms referred to the exact same thing, but one of my clients just requested to have a look at both documents.
The request emanates from the security department of a very big ...
- 630
6
votes
5
answers
18k
views
LVM vs RAID0 vs RAID "linear" - Combine 2 disks as one, data recovery?
given two 2TB USB external disks that have to be combined to one 4TB volume and formatted with one big Filesystem (XFS), I have a small question to ask.
Does LVM provide better Data recovery, should ...
- 271
6
votes
5
answers
471
views
Fault tolerant server structure for the smallest of businesses
I'm trying to figure out what to do for a small business that has been plagued by ridiculous hardware problems. Right now, this business runs on five or six desktop machines; no server infrastructure ...
- 752
6
votes
2
answers
814
views
WHEN to put the contingency plan into action in case of a main server failure?
We have a production SQL Server database server shipping transactional log backups to two standby servers. The disaster recovery plan is already finished: we have a well documented procedure and ...
- 63
6
votes
3
answers
246
views
Disaster Recovery Planning, tower or rack?
I'm working on this project to develop a system with centralized information regarding emergencies delivered via open Wi-Fi on a small city.
I'm from Chile, so we thought of this system to work ...
user78442
6
votes
3
answers
3k
views
Reconstructing .bashrc from running session
I accidentally deleted my .bashrc. I still have the terminal running. What settings can I recover?
I already have the aliases (from the alias command). I assume that all ifs and cases are gone, but I ...
- 93
6
votes
2
answers
269
views
Green System Administrator looking for helpful tips [closed]
I have just been promoted to Systems Administrator for our product. We are designing a application that communicates with the cloud(Amazon EC2). I will be in charge of maintaining all Instances and ...
6
votes
3
answers
3k
views
Is DFSR designed for use for Disaster Recovery?
We are currently working on implementing a DR strategy. Instead of SAN-SAN replication, it has been decided to have 2 live file servers replicating via DFSR. However, I don't know whether or not this ...
- 2,833
6
votes
2
answers
1k
views
Active Directory Disaster Recovery in a Small Business
This is hypothetical question, but one I’m sure that someone must have encountered and/or given some thought to before.
Situation:
Consider this, a small business is running an Active Directory ...
- 321
6
votes
2
answers
745
views
Automated bare-metal recovery practices for small network
I have several machines which are on a small network with one DC and 3 to 5 workstations on the network at any given time. These are all setup with DNS and AD on the same server. I want the ability ...
- 183
6
votes
1
answer
364
views
Recovering lost VHDX / VM (deleted by Veeam)
Earlier this week a VM on one of our hypervisors experienced extended downtime (~24 hours) due to some Windows updates going wrong. I ultimately was able to fix the issue, and noticed yesterday that ...
- 61
6
votes
2
answers
3k
views
MySQL replication issues after a power outage
After a power outage at our data centre, the slave MySQL databases are struggling.
This is in the logs for one of the slaves:
100118 10:05:56 [Note] Slave I/O thread: connected to master 'repl@db1:...
- 335
5
votes
4
answers
400
views
Virtualization for hardware resiliency?
Can anyone tell me if it is possible to pool several physical servers to run a resilient virtualization environment. Our servers are getting more and more critical to our clients and we want to do ...
- 249
5
votes
5
answers
54k
views
Edit Hard Disk Serial Number with VMware
I'm virtualizing a Rockwell AssetCentre Server and I'm looking at Disaster Recovery scenarios. This server contains a lot of other Rockwell Software like RSLinx, Logix 5000, Logix 500, and more...
...
- 459
5
votes
4
answers
555
views
Disaster Recovery/Sabotage Protection for a small business
I've been contacted by two partners in a small professional firm. They are concerned about their other partner and want to take some steps to be absolutely sure that the company's data and systems ...
- 12.9k
5
votes
6
answers
460
views
Can A Virtual Machine Be Converted to a Virtual Server e.g VMWare?
The Simple Question
Can I convert an existing VM to a Virtual Server (e.g. VMWare)? I'm using Oracle's one and only awesome product, VirtualBox, and I'm trying to setup a SharePoint Farm to ...
- 151
5
votes
2
answers
1k
views
Best practice for IIS 6.0 (Windows Server 2003) backups?
What is the best backup strategy for saving IIS 6.0 data: web metadata, files, logs etc. for disaster recovery?
- 28.6k
5
votes
3
answers
2k
views
How can you recover a SQL Server database if the ldf file has been deleted
We had a drive die and lost the ldf file, but the mdf file is in tact. Is there a process for re-connecting to the mdf file, considering the ldf lost? I have searched without much luck.
- 331