6 Resilience and Recovery
Learn how redundancy, protected backups, fault isolation, and tested disaster-recovery planning help systems maintain essential services and recover from disruption.
Building
is built by combining , backups, , and planned recovery rather than relying on any single protective measure. These capabilities serve different purposes: can help maintain service during a component failure, backups support recovery of data or system state, and isolation helps contain failures.
and availability
provides alternate components or paths so that one failure does not stop the whole service. Examples include application instances in separate availability zones, more than one network path, and a standby database. Load balancing and health checks can route requests away from failed components, while automated failover can promote a standby system.
is most useful when it avoids common points of failure. Two servers in the same rack might fail together if they depend on the same power supply, network switch, or database. Distribute critical components across suitable failure domains and identify shared dependencies.
can improve availability, but it does not replace backups. Replicated systems can also replicate accidental deletion, corruption, or malicious changes.
Protecting and testing backups
Decide what data and system state must be backed up, how often copies should be made, how long they should be retained, and where they should be stored. Protect copies with encryption, restricted access, and separation from ordinary production credentials and systems. Where appropriate, immutable or offline copies can reduce the chance that an attacker or compromised administrator can alter or delete every copy.
A is useful only if it can be restored. Periodically restore data and verify its completeness, consistency, and usability; record the restoration procedure and results. Replication can reduce data-loss exposure, but pair it with point-in-time or versioned backups when recovery from corruption or destructive changes is important.
Containing failures
confines a failure to a defined boundary so unaffected parts of the system can continue operating. Separate services, workloads, or customers into appropriate isolation boundaries. Resource limits, timeouts, health checks, and controlled retries can help prevent a failing or overloaded component from exhausting shared resources.
This approach is sometimes called a : like compartments in a ship, independent sections limit how far damage spreads. For example, if a reporting service becomes slow, a separate resource pool and request limit can prevent it from consuming all the database connections needed by checkout.
Isolation boundaries should account for dependencies, not only application servers. A shared database, identity service, or network can still become a failure point.
Planning for disaster recovery
A documents how to restore technology services after a major disruption. Begin with a business impact analysis: identify critical services and their dependencies, the consequences of downtime or data loss, and the order in which services must be recovered.
Assign people responsible for decisions and actions, and coordinate the DR plan with business-continuity and incident-response plans. Contingency planning is a coordinated process that includes impact analysis, recovery strategies, a plan, exercises, and ongoing maintenance.
Setting recovery objectives
Set measurable recovery objectives based on business needs:
: the maximum acceptable time to restore a service.
: the maximum acceptable amount of data loss, expressed as the time between the latest recoverable data and the disruption.
For example, an RTO of four hours and an RPO of fifteen minutes means the service should be restored within four hours, using data no more than fifteen minutes older than the disruption. Choose a recovery approach that can meet these objectives and is practical to operate.
Choosing a recovery approach
Recovery approaches trade cost and complexity against recovery time and potential data loss. Choose an approach according to business impact rather than assuming every service needs the fastest, most expensive option.
and restore: restore data and rebuild services after a disruption. This approach is usually simpler and less costly, but slower.
Pilot light or warm standby: keep core systems or a reduced-capacity environment ready, then scale it or complete deployment during recovery.
Active-active: operate in multiple locations and continue serving users if one location fails. This can reduce recovery time, but adds cost and complexity.
Testing and maintaining the plan
A plan must be tested, not merely documented. Exercise restoration, failover, communications, access to recovery credentials, and dependencies. Verify that data is usable and the recovery environment is current.
Record gaps, assign corrective actions, and retest after significant system or organizational changes. Plan how to fail back safely to normal operations, including how to reconcile data created during recovery.