Alauda Cache Service E1

Failover

When the upstream Redis instance fails, you need to perform a disaster recovery failover for the replication group first, and then switch the client's access address to the promoted instance.

Applies to Disaster Recovery mode

This procedure applies to Disaster Recovery mode (spec.activeRedis.mode: peerof) on both Redis 6.0 and Redis 7.2, where replication is directed and one instance is the upstream.

Active-Active mode has no failover step of this kind: every member is already active and accepts writes, so recovering from the loss of a datacenter is a matter of redirecting clients to a surviving member. Removing the lost member from the mesh is described in Removing a member.

TOC

Replication Group SwitchFailover Timing DescriptionDisaster Recovery SwitchUse the failed upstream as a downstream of the new upstreamClient-side Disaster Recovery Switch

Replication Group Switch

In a disaster recovery group, when the upstream fails, the synchronization link with the failed upstream needs to be interrupted, and the downstream side is promoted to the upstream; to ensure that the client can write data and that the dirty data of the failed upstream will not pollute the downstream instance.

Failover Timing Description

When a failure occurs at the upstream of the replication group, Alauda Cache Service E1 on the downstream side will detect that the link with the upstream is interrupted. At this time, the status of the ActiveRedisConnection resource will be abnormal, and the Web Console of the downstream instance will prompt that a disaster recovery switch can be performed.

Disaster Recovery Switching Risk Description
It should be noted that this abnormal prompt is only an interactive alarm prompt and should not be used as the sole basis for judging that the client can perform a disaster recovery switch.

Disaster Recovery Switch

CLI
Web Console

View Connection Status

# View normal status
$ kubectl -n default get activeredisconnections
NAME           INSTANCE   STATUS    MESSAGE   AGE
c6-dest-conn   c6-dest    Healthy             35s

# View abnormal status
$ kubectl -n default get activeredisconnections
NAME           INSTANCE   STATUS   MESSAGE                                                                                          AGE
c6-dest-conn   c6-dest    Failed   shard 0 status is Disconnected; shard 1 status is Disconnected; shard 2 status is Disconnected   86s

When STATUS is Failed, it indicates that the connection between instance c6-dest and the upstream is abnormal. You can view the yaml details to understand the specific situation:

$ kubectl -n default get activeredisconnections c6-dest-conn -o yaml
apiVersion: redis.middleware.alauda.io/v1alpha1
kind: ActiveRedisConnection
metadata:
  annotations:
    cpaas.io/creator: admin
    cpaas.io/updated-at: "2025-08-27T08:42:16Z"
  creationTimestamp: "2025-08-27T08:42:16Z"
  generation: 1
  labels:
    cpaas.io/activeredis: c6-dest-activeredis
    cpaas.io/activeredis-instance: c6-dest
  name: c6-dest-conn
  namespace: default
  ownerReferences:
  - apiVersion: redis.middleware.alauda.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ActiveRedis
    name: c6-dest-activeredis
    uid: ed1010f8-a70d-42ef-be19-0450478df137
  resourceVersion: "19592690"
  uid: 301f77be-ff52-4e50-98d0-d85a38e2415a
spec:
  addresses:
  - 192.168.1.11:30011
  instance: c6-dest
  pause: false
  teardownPolicy: Detach
status:
  instance: c6-dest
  message: shard 0 status is Disconnected; shard 1 status is Disconnected; shard 2
    status is Disconnected
  shards:
  - index: 0
    offset: "0"
    opId: "0"
    status: Disconnected
    syncStatus: PartialSync
  - index: 1
    offset: "0"
    opId: "0"
    status: Disconnected
    syncStatus: PartialSync
  - index: 2
    offset: "0"
    opId: "0"
    status: Disconnected
    syncStatus: PartialSync
  status: Failed
  upstreamPeer:
    service_id: 0
    service_metadata:
      instance: default/c6

Disconnect from the upstream

After manually confirming that the upstream is indeed abnormal, you need to disconnect the downstream instance from the upstream link to prevent the introduction of dirty data.

$ kubectl -n default delete activeredisconnections c6-dest-conn

After deleting the corresponding ActiveRedisConnection resource, the Redis instance has become an independent upstream instance, and the client's access address can be safely switched to this instance for reading and writing.

Keep the default Detach teardown policy for a failover

Deleting the connection tears the link down according to its spec.teardownPolicy. Use the default Detach so the failed datacenter can rejoin later. Reserve Decommission for a datacenter that is gone for good — it is irreversible and forces a full resynchronization if the peer ever returns. See Removing a connection.

Use the failed upstream as a downstream of the new upstream

After the failed upstream returns to normal, it can be re-added to the replication group as a downstream instance of the promoted upstream. For the operation method, refer to Set Up Disaster Recovery Replication.

Client-side Disaster Recovery Switch

Before connecting to the replication group, the client should support switching the Redis access address to the promoted instance after detecting the upstream failure. Usually, there are the following switching methods:

ArchitectureMechanismAdvantagesDisadvantagesImpact on RTO/RPOImplementation DifficultyIdeal Use Case
DNS SwitchUpdate DNS records to point to the new IPSimple concept, platform-independent.Long and uncontrollable RTO (depends on TTL), invalid for K8s internal traffic.RTO: minutes
RPO: may be high.		LowApplications with low RTO requirements, or as a manual fallback solution.
ProxyThe client connects to the proxy, and the proxy routes to the master node through health checks.Fast switching (second-level RTO), transparent to the client, simple fault recovery.Increases network hops and operation and maintenance burden, the proxy itself needs to be highly available.RTO: seconds
RPO: low.		MediumRecommended solution: requires fast and transparent switching, and has the ability to operate and maintain a highly available proxy cluster.
Service Mesh (Istio)The Sidecar proxy intercepts traffic and performs local priority and cross-cluster switching based on policies.Powerful functions, application transparency, simple fault recovery.Extremely high operation and maintenance complexity, heavy technology stack.RTO: seconds
RPO: low.		HighLarge and complex systems that have fully adopted service mesh to manage microservices.
Client LibraryThe library has built-in logic and decides to switch to the other cluster by itself.No middle layer, latency may be lower.Extremely high risk: switching decisions are unreliable, fault recovery is complex (often requires restarting the business), and ecological support is inconsistent.RTO: unpredictable
RPO: high risk.		MediumNot recommended for production-level automatic disaster recovery.

Whether the client triggers a disaster recovery failover cannot be judged solely by a binary decision; failover is a multi-dimensional, high-confidence decision-making process. It needs to be handled in combination with multiple dimensions of fault detection, including but not limited to: the available status of the instance, whether the instance still has high availability, whether the k8s cluster has the possibility of continuing to serve, data center availability detection.

Usually, the following expression needs to be met before the client can safely perform a disaster recovery switch:

(Manual trigger switch)
OR
(
  (Upstream instance status check fails, and continues to fail for N times)
  AND
  (
    (The instance no longer has high availability)
    OR
    (The data center availability check fails, and continues to fail for N times)
  )
  AND
  (The downstream instance availability status check passes)
)
Support Description

Alauda Cache Service E1 currently does not provide support for client-side disaster recovery switching. Customers need to implement a suitable client switching method according to their own infrastructure.