Active-Active DR: Recovery, Reconciliation, and Failback

The first post in this series argued that the hard part of Kafka-based Active-Active is write ownership, not cluster replication. The second post showed how to enforce that ownership across a Home Region write path.

Now assume that Region A actually fails.

The application in Region B is alive and still serves reads. An operator redirects traffic to B and transfers write ownership. The dashboard turns green again.

Is recovery complete?

• Did the last database commit in A reach B?
• How far did B’s business database apply data, not merely Kafka replication?
• Did a timed-out payment request fail or succeed?
• When can the recovered Region A be trusted again?
• Can failback reuse A’s old owner epoch?

If the system cannot answer those questions, failover may be complete while recovery is not.

Failover changes where writes happen.
Recovery proves which data survived.
Reconciliation resolves the remaining uncertainty.
Failback is another ownership transfer.

This final post in the Active-Active DR series closes the operational path from incident detection through data reconciliation and failback.

Here, failover does not mean waking an idle standby. Both regions already serve traffic, but only one writer exists for a given ownership key. During an incident, the system transfers write ownership for the affected keys and shifts the failed region’s workload to the survivor.

Traffic Routing and Write Ownership Are Separate States

A global load balancer decides which region receives a user request. The ownership store and write boundary decide which region may modify a particular ownership key.

traffic route: Region A + Region B -> Region B only
write owner:   Region A · epoch 42 -> FENCED -> Region B · epoch 43

Those states do not necessarily change together. If health checks remove A while ownership still points to A, B must not grant itself write permission. Conversely, if ownership has moved to B while some requests still reach A, A is a forwarding or rejection point, not the new writer.

DR automation must therefore record traffic and ownership separately, verifying both successful writes in the target and rejection of the previous epoch.

Separate a Successful Switch from a Successful Recovery

A global load balancer sending traffic to B proves that the network path changed. A few successful requests in the new region do not prove that the entire service and its data recovered.

At least five dimensions belong in the recovery criteria.

Dimension	Question	Required evidence
Availability	Did critical functions return within the target time?	Actual time from first user impact or incident start to verified service restoration
Ownership safety	Is the previous region unable to write?	Current owner and epoch, stale-writer rejections, database or write-boundary fencing state
Data recovery	Through which point did the target apply data continuously?	Contiguous checkpoints across Outbox, cross-region replication, Inbox, and business database
Capacity and dependencies	Can the target sustain the full load and essential external functions?	Load tests and real calls to hard dependencies such as identity, KMS, DNS, and payment systems
Stabilization	Were missing, duplicate, conflicting, and unknown-outcome operations classified?	Reconciliation ledger, decisions, approvals, and audit history

Measure RTO until essential business functions return in a verified state, not merely until traffic routing changes. Calculate RPO from the checkpoint observed during the incident, not from a normal replication-lag setting such as three seconds.

The Google Cloud disaster recovery planning guide treats RTO and RPO as measurable objectives and notes that a time-based RPO does not describe the amount or quality of lost data. Three payment approvals and three cache refreshes can fall within the same three-second interval but have very different business impact.

Execute Failover as a State Machine

A list of commands does not explain which conditions permit the next recovery step. It is safer to model failover as a state machine that moves ownership and data evidence together.

Each state should answer a concrete question.

1. DETECT: Did the system combine user errors, regional dependencies, and data-layer health instead of reacting to one failed health probe?
2. DECLARE: Is the scope a whole region, one service, or a range of ownership keys, and is this a planned or forced transfer?
3. FENCE A: Does credential revocation, database read-only mode, lease expiry, or conditional epoch validation block writes at a boundary the old writer cannot bypass?
4. CHECKPOINT: What is the last version applied continuously in B, and which interval remains unknown?
5. ACTIVATE B: Did ownership move to a new epoch and begin with a limited set of writes?
6. RECONCILE: Were missing changes, duplicates, conflicts, and external side effects classified?
7. STABLE B: Were actual RTO and RPO recorded and steady state proven under full load?
8. PLAN FAILBACK: Was recovered A rebuilt like a new replica before another planned transfer?

A planned transfer can prove FENCED(A) -> CHECKPOINT -> ACTIVE(B) in order. A forced transfer after complete isolation of A may be unable to prove fencing or the final source commit.

Do not convert “cannot confirm” into “safe.” Keep uncertainty in the recovery state with fields such as mode=forced and rpo_status=unknown. The incident commander must then choose whether to keep writes closed for consistency or accept a defined data risk and activate B.

Not Every Incident Should Trigger Failover

An ownership transfer can be appropriate when regional infrastructure is unreachable or the current writer cannot serve business requests. If the cause is a faulty deployment, mass deletion, or data corruption, however, replication can carry the same damage rapidly into the other region.

regional outage       -> fence old writer + transfer ownership
logical corruption    -> freeze affected writes + choose clean point + restore/replay
credential compromise -> revoke access + rotate credentials + audit the scope

Immediately activating B during logical corruption can simply serve the same damaged state at higher throughput. Freeze writes for the affected scope, restore a clean point from versioned backups or point-in-time recovery, and replay only verified events.

The AWS Well-Architected guidance for DR strategies likewise notes that continuous replication can help with some disasters but does not protect against data corruption or destruction without versioning and point-in-time recovery. The DECLARE state should retain incident_kind as well as transfer scope.

Kafka Lag Alone Cannot Define the Recovery Point

A typical Home Region change travels through this path.

Region A DB commit
-> Outbox record
-> Kafka publish
-> cross-region replication
-> Region B Inbox
-> Region B business DB apply

Kafka consumer lag reaching zero does not prove end-to-end recovery.

• A stopped Outbox relay can leave committed database changes that never entered Kafka.
• A replication link disconnected from its source can stop reporting new lag without proving that no new events existed.
• Events may have reached Kafka while B’s Inbox or business database remains behind.
• Partition offsets show delivery positions, but they do not prove contiguous business versions for an aggregate.

Instead of listing independent pipeline numbers, express the recovery point as the highest contiguous business version applied and verified in B’s business database.

published watermark    = last point confirmed from A's Outbox into Kafka
replicated watermark   = last point confirmed in B's Kafka
applied watermark      = last contiguous point atomically committed by B's Inbox and business DB
recovery checkpoint    = applied watermark verified by business invariants in B

The design does not need one global sequence for every entity. It can retain versions per aggregate_id or ownership key, offsets per Kafka partition, and event timestamps. What matters is the ability to calculate which unit is safe through which point during an incident.

Until A is recovered, a final commit may exist only in its source database. The checkpoint therefore proves “the last safe point in B,” not “exactly zero records were lost after it.” Actual RPO can be finalized after residual data and logs from A are compared; if they can never be recovered, the unverified interval must remain explicit.

A Runbook Must Retain State and Evidence

A useful runbook does more than list actions such as “change DNS.” It records current state, transition conditions, results, and approvers in a machine-readable form.

incidentId: inc-2026-0716-01
incidentKind: regional-outage
scope:
  type: ownership-key-range
  value: tenant-0001..tenant-2499
mode: forced
sourceRegion: region-a
targetRegion: region-b
fromEpoch: 42
toEpoch: 43
oldOwnerFenced: unconfirmed
recoveryCheckpoint:
  type: aggregate-version
  value: order:812
rpoStatus: unknown
approval:
  dataRiskAcceptedBy: incident-commander
  approvedAt: 2026-07-16T07:18:22Z
state: ACTIVATE_B

Every action should be idempotent, and each state transition should use compare-and-set semantics to verify the previous state. Connect ownership records, checkpoints, and validation results to one incident record instead of leaving only operator notes and script output.

Automation also needs boundaries.

• Collecting incident evidence, checking dependencies, calculating checkpoints, and validating target capacity are strong automation candidates.
• Executing the already-approved steps of a planned transfer is also suitable for automation.
• A forced takeover without proven fencing or RPO carries data-loss risk and needs an explicit approval gate.
• Operators need an abort mechanism when automation behaves unexpectedly.

The AWS Well-Architected guidance for recovery automation similarly recommends observable and repeatable automation while warning against automatically executing high-data-risk procedures or omitting a way to halt a dangerous recovery.

Do Not Open the Entire Target Region at Once

Region B serving half of the normal read traffic does not prove that it can absorb all writes and the full workload from A. Validate more than the application replica count.

database connections and write IOPS
Kafka producer and consumer throughput
Outbox and Inbox backlog drain rate
cache warm-up and rate limits
KMS, secrets, identity, and permissions
payment, email, object storage, and other external dependencies
operator and deployment-pipeline access to Region B

When possible, activate the new epoch for a small range of ownership keys first. Do not stop after one synthetic write returns successfully. Verify the complete result.

1. B committed its business row and Outbox record in one transaction.
2. The event reached B’s Kafka and every required cross-region path.
3. Inbox processing and projections applied the expected version.
4. A read-back satisfies the business invariants.
5. Retrying the same idempotency key causes the external effect only once.
6. Requests carrying A’s stale epoch continue to be rejected.

Then expand the ownership-key range and traffic weight gradually. Watch stale-epoch rejection, oldest Outbox age, Inbox apply lag, database saturation, and external API failures alongside the ordinary error rate.

Failover Creates a Data Reconciliation Queue

A forced transfer does not remove uncertainty. It draws a boundary that prevents uncertainty from growing. When A returns, collect its records as evidence instead of copying its database wholesale over B.

A practical classification can look like this.

Class	Example decision	Handling principle
Duplicate	The same event_id and payload hash already exist in B’s Inbox	Do not apply again; record the comparison
Missing	Aggregate versions contain a gap such as 41, 42, 44	Recover the original event, replay idempotently, and verify invariants
Conflict	The same aggregate version has a different payload hash or business state	Quarantine it and apply a business decision instead of hiding it with last-write-wins
Side effect	A payment approval or email delivery cannot be established from the local database	Query the external system by idempotency key before retrying or compensating

In particular, do not copy a row discovered under an old epoch directly into B. First determine whether it is a valid missing commit, a stale write produced through an unfenced path after the transfer, or state already superseded by another event.

The reconciliation ledger should retain at least event_id, ownership key, aggregate version, owner epoch, payload hash, both observed states, classification, replay or compensation result, and approver. This connects the actual data impact and its resolution to the incident report.

Failback Is a New Ownership Transfer, Not an Undo

Do not automatically restore A’s former role when it starts responding again. After a forced transfer, B is the current source of truth and A is a stale replica.

A safe failback resembles this sequence.

1. Keep every write path in recovered A fenced.
2. Rebuild A from B's snapshot and events.
3. Verify that A caught up continuously through B's checkpoint.
4. Confirm that unresolved reconciliation items are below the transfer threshold.
5. Start a new planned ownership transfer from B to A.
6. Fence B and issue new epoch 44 to A.
7. Activate A for a small scope and prove steady state again.

Never reuse A’s former epoch 42. Reuse would make requests processed under B’s epoch 43 indistinguishable from delayed requests left in A. Failback is a separate transfer with another monotonically increasing epoch.

If both regions are symmetric and B can continue normal operation, immediate failback may not be necessary. Decide based on cost, data locality, latency, and capacity for the next failure, not merely because A used to be primary.

The Azure Well-Architected disaster recovery guidance recommends treating failback as a distinct process with the same level of planning and validation as failover. It should not be modeled as the failover commands run in reverse.

DR Drills Measure Recovery Capability, Not Architecture

Having a DR document does not prove that the team can recover within its objectives. The AWS Well-Architected guidance for DR testing recommends regularly exercising the recovery site and verifying that actual RTO and RPO are met.

Do not rehearse only one “entire region is down” scenario. Split drills across the boundaries where the design is most likely to fail.

Drill scenario	Required result
Complete loss of Region A	B can activate without A’s control plane, and operators can access required credentials and runbooks
Network partition between A and B	Both regions cannot write the same ownership key; stale epochs are rejected
Ownership Control Plane outage	Existing ownership or the intended fail-closed policy remains effective
Outbox or cross-region replication delay	The decision uses the applied checkpoint instead of one Kafka lag number
Logical data corruption	The team avoids a blind regional switch, selects a clean recovery point, and performs restore plus verified replay
Partial dependency failure in B	Health checks do not hide failures in hard dependencies such as identity, KMS, or payment systems
Full-load operation in B	Database, Kafka, cache, and external APIs remain stable at the target throughput
Failback	A is rebuilt while B remains authoritative, followed by a planned transfer using a new epoch

Exercises can progress through tabletop review, non-production dry runs, limited production ownership-key transfers, and full regional transfers. The important point is not to declare success after reading a document while the realistic recovery path remains unexecuted.

Record these measurements after each drill.

failure detection time
DR declaration and approval time
old-owner fencing completion time
checkpoint decision time and evidence
first verified write time
full-load transfer time
actual RTO and actual RPO
counts of missing, duplicate, conflicting, and unknown-outcome operations
number of manual steps and failed automation actions
steady-state and failback completion time

After a substantial architecture or dependency change, retest instead of treating an old drill result as current recovery capability.

The Operational View Must Connect the Recovery State

During an incident, operators do not have time to mentally join separate Kafka dashboards, database consoles, load balancers, and ownership stores. At minimum, connect this evidence by incident and ownership scope.

• Current recovery state and time of its last transition
• Source and target regions, owner epoch, and fencing evidence
• Oldest Outbox age and publish and cross-region replication watermarks
• Inbox backlog and business-database applied checkpoint
• Target capacity and hard-dependency health
• Canary results, stale-writer rejections, and unknown-outcome requests
• Reconciliation classes and unresolved counts
• Approvers, automation history, aborts, and retries

An operations platform should therefore do more than mark individual resources as healthy. It should connect the evidence required for a recovery decision into one workflow. Konduo is designed around that goal: connecting resource state, metric and log evidence, diagnostics, alert response, and operational history in one integrated operations platform.

Final Review Checklist

• Can DR scope be expressed by ownership key or business boundary, not only as an entire service?
• Are traffic-routing state and write-ownership state recorded and validated separately?
• Does the declaration distinguish regional isolation, logical corruption, and credential compromise?
• Does RTO end at verified critical-function recovery rather than at a traffic change?
• Can RPO be calculated from an end-to-end checkpoint across Outbox, Kafka, Inbox, and the business database?
• Can the system represent unknown and retain the approver when the source is unreachable?
• Can the previous owner be fenced at a write boundary it cannot bypass?
• Does forced-transfer automation include an approval gate and abort mechanism?
• Has the target region actually proven full-load and hard-dependency capacity?
• Can a subset of ownership keys move as a canary?
• Is there a reconciliation ledger for missing data, duplicates, conflicts, and external side effects?
• Does the process avoid overwriting the current owner with data from the recovered old region?
• Have versioned backups and point-in-time recovery been exercised under the assumption that replicated data can also be corrupted?
• Is failback a separate planned transfer that issues a new epoch?
• Do recurring drills record actual RTO, RPO, and reconciliation results?

Conclusion

Active-Active DR is not complete when applications and Kafka are deployed in two regions. Operational work remains even after Home Region writes are enforced and traffic moves to the surviving region.

The system must prove that the old writer is fenced. It must declare the continuous point through which the new region applied data. It must reconcile missing data, duplicates, conflicts, and external side effects. When the recovered region returns, it must receive another ownership transfer instead of reclaiming its past state.

Failover is the starting point.
A verified checkpoint and reconciliation complete recovery.
Failback with a new epoch establishes the next steady state.

Seen this way, the final artifact of Active-Active DR is not an impressive multi-region topology diagram. It is a recovery state machine the team can execute and measure repeatedly, together with an operational record that preserves data uncertainty instead of hiding it.

Further Reading

• Kafka Active-Active Is About Ownership, Not Kafka - Starts the series with the write-ownership model that must precede cluster topology.
• Home Region Writes: Keeping Database and Kafka Ownership Aligned - Connects ownership lookup, database commits, Outbox, Kafka, and epoch fencing into one write path.
• Why Kafka Fits Between Outbox and Inbox - Explains the producer and consumer idempotency contract that makes reconciliation possible.