#Active-Active Operations

Running an Active-Active replication group after it is set up: keeping the member clocks in agreement, adjusting the gossip and membership timers, and adding or removing members. For the setup procedure see Setup; for why these things matter see Architecture.

#Clock synchronization

#Why the clocks matter

Both of the effects below are proportional to how far the clocks are apart, and neither produces an error at the time it happens.

• Conflict resolution is decided by wall-clock time. The last-write-wins rule compares a Hybrid Logical Clock whose physical component is the local wall-clock millisecond. A member whose clock runs ahead of the others therefore wins conflicts it should have lost: a value written later in real time on another member is silently overwritten by the earlier write from the fast member. Every last-write-wins command in Command Support is affected — including the per-field resolution used by hashes and sorted sets.
• Deleted data can come back. A deletion is retained as a tombstone until every active member has acknowledged it and the tombstone is at least five minutes old. That age floor is measured in wall-clock time. Once skew approaches half of it, a tombstone can be released on one member while a concurrent addition from a skewed member is still in flight, and the deleted element reappears there. This is a genuine convergence break, not a transient inconsistency.

#How the difference is measured

• At the peer-port handshake, members exchange four timestamps and compute a round-trip-corrected offset — the same calculation NTP uses. Of the samples taken for a peer, the one with the shortest round trip is kept, and it is replaced when it goes stale, so queueing delay does not bias the reading.
• Between handshakes, gossip refreshes a coarser offset for every member roughly once per gossip round.
• About once per second the module takes the largest absolute offset across all members as the current skew, and evaluates the two alarm tiers against it.

An offset is always reported as remote clock − local clock in milliseconds, relative to the member you read it from. A positive value means the listed member's clock is ahead of the one you are reading.

#Alarm tiers

An alarm clears on its own, with hysteresis, once the skew falls below 0.8 times its threshold. No restart is needed.

#How to check the difference

Start from the resource, which reports the offset to every member the local instance knows about:

$ kubectl -n default get activeredismesh mesh-dc1 \
    -o jsonpath='{range .status.members[*]}{.address}{"\t"}{.state}{"\t"}{.clockOffsetMs}{"\n"}{end}'
redis-dc1.example.com   alive   0
redis-dc2.example.com   alive   -5
redis-dc3.example.com   alive   4180

For the aggregated gauges and the attribution flag, read them on the instance itself:

$ redis-cli INFO activeredis | grep -E \
    'mesh_max_clock_skew_ms|current_max_skew_ms|peak_skew_ms|clock_skew_violations|clock_skew_critical_violations|local_clock_is_outlier'

Per-member offsets are also available directly from the module, as clock_offset_ms on each line:

$ redis-cli as.mesh nodes

#How to confirm that clock skew caused a problem

1. Confirm an alarm actually fired. clock_skew_violations or clock_skew_critical_violations is non-zero, and the Redis log carries the matching line. The advisory line begins clock-skew advisory: and the critical line begins CRITICAL clock-skew (resurrection risk):; both name the offending member's svc= (its serviceID) and the measured offset. A problem that predates any alarm is not a skew problem.
2. Identify which clock is wrong. If local_clock_is_outlier is 1, this node's clock is the outlier and this node is the one to fix. Otherwise the wrong clock belongs to the member with the largest clockOffsetMs.
3. Check that attribution is even possible. Blame can only be assigned with at least three members. In a two-member group the module can report only that the pair disagrees; compare both nodes against an external time reference instead.
4. Match the symptom to the tier. A value that was written later in real time but lost to an earlier write matches the advisory tier. An element that was deleted and later reappeared on one member matches the critical tier — check whether clock_skew_critical_violations rose in the same window.
5. Do not read hlc_drift_rejected as a skew measurement. That counter, in the crdt section of INFO activeredis, counts writes that lost the timestamp comparison. It rises during ordinary concurrent writes as well. It is a corroborating symptom when it climbs alongside a non-zero current_max_skew_ms, nothing more — current_max_skew_ms is the measurement.

Once the offending clock is corrected the alarm clears by itself. Data that was already resolved the wrong way is not re-resolved: a value lost to a skewed comparison stays lost, and a resurrected element stays resurrected until it is deleted again.

#Tuning

spec.tuning pushes optional gossip and membership timers to every node.

$ kubectl -n default patch activeredismesh mesh-dc1 --type=merge \
    --patch='{"spec": {"tuning": {"gossipIntervalMs": 500}}}'

#Managing membership

#Adding a member

Enable Active-Active mode on the new instance, then create its ActiveRedisMesh with the existing members as seeds — see Setup. The new member joins and is discovered by the whole mesh through gossip — existing members do not need their seed lists updated for the mesh to converge. Updating them is still recommended, so that a member that restarts while the original seeds are unreachable can still rejoin.

#Removing a member

1. Remove the departing member's address from the surviving members' spec.seeds.
2. Delete the departing member's ActiveRedisMesh resource, and then its Redis instance.

#Limitations

Active-Active mode inherits the constraints listed in Limitations and Risks, including matching architectures and shard layouts across members, the [0-15] serviceID range, and RDB-only persistence.

Because every member accepts writes, Active-Active exercises the conflict-resolution rules continuously. Read Command Support before pointing an application at an Active-Active group — it lists which commands are refused, which run locally without replicating, and how each replicated command resolves a conflict.