Team of professionals

Back to all news

Disaster Recovery in Ceph with cephadm, Ceph-CSI, and RBD Mirror

Introduction

Ceph is a highly available, scalable, and resilient storage solution widely used in cloud and enterprise environments. However, even with its built-in redundancy, disaster recovery (DR) strategies are essential to ensure business continuity in case of data center failures, network outages, or hardware failures. Ceph provides robust disaster recovery options, including RBD mirroring, to replicate block storage volumes across geographically separated Ceph clusters.

With the introduction of cephadm, Ceph cluster management has become more straightforward, making it easier to deploy and maintain disaster recovery setups. Additionally, Ceph-CSI enables Kubernetes clusters to consume Ceph storage efficiently. In this article, we will explore how to set up disaster recovery in Ceph using cephadm, Ceph-CSI, and RBD Mirror to protect RBD volumes used by Kubernetes clusters deployed across two data centers.

Disaster Recovery Architecture

We have two geographically separated data centers:

  • Primary Data Center (Production): Hosts a Kubernetes cluster and a Ceph cluster.
  • Secondary Data Center (Disaster Recovery – DR): Hosts another Kubernetes cluster and a Ceph cluster where data is replicated using RBD mirroring.

Kubernetes workloads in the primary DC store their persistent data in Ceph RBD volumes via Ceph-CSI. These volumes are mirrored asynchronously to the secondary DC using RBD mirroring, ensuring data availability in case of a failure in the primary DC.

Deploying Ceph with cephadm in Both Data Centers

Bootstrap the Ceph Cluster

On each Ceph cluster (Primary and Secondary):

cephadm bootstrap --mon-ip

Add Additional Nodes

cephadm shell -- ceph orch host add

Deploy Required Services

ceph orch apply mon
ceph orch apply mgr
ceph orch apply osd --all-available-devices
ceph orch apply rbd-mirror

Ensure that the rbd-mirror daemon is running on both clusters:

ceph orch ps | grep rbd-mirror

Configure RBD Mirroring

On the primary Ceph cluster:

rbd mirror pool enable  snapshot

Export and import the authentication key:

ceph auth get-key client.rbd-mirror > rbd-mirror.key
scp rbd-mirror.key 
ssh  'ceph auth import -i rbd-mirror.key'

On the secondary Ceph cluster, add a peer connection:

rbd mirror pool peer add  client.rbd-mirror@

Verify peering status:

rbd mirror pool status

Installing Ceph-CSI in Kubernetes Clusters

Now we have ceph cluster ready and we can deploy ceph-csi on our k8s clusters. We need to deploy ceph-csi in both locations, but

Deploy Ceph-CSI Driver

kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/devel/deploy/csi-rbdplugin.yaml
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/devel/deploy/csi-rbdplugin-provisioner.yaml

Enable RBD Mirroring on the Pool

rbd mirror pool enable  snapshot

Configure StorageClass to Use the Mirrored Pool

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd-mirrored
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: 
  pool: 
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
  csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: true

Apply this StorageClass:

kubectl apply -f storageclass.yaml

Create a PersistentVolumeClaim (PVC)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: ceph-rbd-mirrored

Failover Process (Switching to the Secondary Data Center)

Promote the Secondary Ceph Cluster

rbd mirror pool promote

Update ClusterID and PoolID Mappings

Ensure that the Kubernetes cluster in the DR site correctly maps the Ceph cluster’s ClusterID and PoolID using the predefined mapping.

       apiVersion: v1
       kind: ConfigMap
       metadata:
         name: ceph-csi-config
       data:
         cluster-mapping.json: |-
           [
             {
               "clusterIDMapping": {
                 "primary-cluster-id": "secondary-cluster-id"
               },
               "RBDPoolIDMapping": [
                 {
                   "1": "2"
                 },
                 {
                   "11": "12"
                 }
               ]
             }
           ]

Apply this updated mapping:

      
kubectl apply -f ceph-csi-config.yaml

Modify Ceph-CSI Config to Update Monitor Addresses on Secondary Cluster

To use a mirrored and promoted RBD image on a secondary site during a failover, you need to replace the primary monitor addresses with the IP addresses of the secondary cluster in ceph-csi-config. Otherwise, Ceph-CSI won’t be able to use the volumes, and application pods will become stuck in the ContainerCreating state. Thus, during failover, both clusters will have the same monitor IP addresses in csi-config on secondary site.

       apiVersion: v1
       kind: ConfigMap
       metadata:
         name: ceph-csi-config
       data:
         config.json: |-
           [
             {
              "clusterID": "ceph1",
              "rbd": {
                 "radosNamespace": "",
              },
              "monitors": [
                "192.168.39.82:6789"
              ],
              "cephFS": {
                "subvolumeGroup": ""
              }
             },
             {
              "clusterID": "ceph2",
              "rbd": {
                 "radosNamespace": "",
              },
              "monitors": [
                "192.168.39.82:6789"
              ],
              "cephFS": {
                "subvolumeGroup": ""
              }
             }
           ]

Apply the updated configuration

kubectl apply -f ceph-csi-config.yaml

Modify StorageClass to Point to the Secondary Cluster

parameters:
clusterID: secondary-cluster-id

Apply the modified StorageClass:

kubectl apply -f storageclass.yaml

Restart Affected Workloads

kubectl rollout restart deployment

Validate Data Accessibility

Ensure the applications can access data stored in the secondary Ceph cluster.

Failback Process (Restoring to the Primary Data Center)

Demote the Secondary Cluster and Re-enable Mirroring

rbd mirror pool demote

Update ClusterID and PoolID Mappings Back to Primary

apiVersion: v1
kind: ConfigMap
metadata:
  name: ceph-csi-config

Modify StorageClass to Point Back to the Primary Cluster

parameters:
clusterID: primary-cluster-id

Restart Workloads to Use the Primary Cluster

kubectl rollout restart deployment

Verify Mirroring and Data Integrity

rbd mirror pool status

Conclusion

By configuring ClusterID and PoolID mappings and ensuring proper Ceph monitor address updates during failover, you enable seamless disaster recovery for Kubernetes workloads using Ceph-CSI. This approach maintains data accessibility and consistency, facilitating a smoother failover and failback process. Using cephadm, deploying and managing mirroring has become significantly easier, enabling organizations to set up failover mechanisms efficiently. By following the above steps, you can ensure data integrity, minimize downtime, and enhance business continuity in the event of a disaster.

Author

Kamil Madáč
Grow2FIT Cloud&DevOps Consultant

Kamil is a Senior Cloud / Infrastructure consultant with 20+ years of experience and strong know-how in designing, implementing, and administering private cloud solutions (primarily built on OpenSource solutions such as OpenStack). He has many years of experience with application development in Python and currently also with development in Go. Kamil has substantial know-how in SDS (Software-defined storage), SDN (Software-defined networking), Data storage (Ceph, NetApp), administration of Linux servers and operation of deployed solutions.
Kamil regularly contributes to OpenSource projects (OpenStack, Kuryr, Requests Lib – Python).

The entire Grow2FIT consulting team: Our Team

Related services