How to Prepare a Dev/Test Ceph Environment

When you are using Ceph in production, it is important to have an environment where you can test your upcoming upgrades, configuration changes, integration of new clusters, or any other significant changes without touching real production clusters. Such an environment can be built with a tool called Vagrant, which can very quickly build a virtualized environment described in one relatively simple config file.

We use Vagrant on Linux with the libvirt and hostmanager plugins. Libvirt is a toolkit to manage Linux KVM VMs. Vagrant can also create virtualized networks to interconnect those VMs and storage devices, so you can have an almost identical copy of your production cluster if you need it.

The setup

Let's create a 5-node Ceph cluster. The first three nodes will be dedicated to control-node daemons, all nodes will also be OSD nodes (2 × 10 GB disks on each node by default), and one node will be a client node. Client nodes can be used for testing access to cluster services — mapping RBD images, mounting CephFS filesystems, accessing RGW buckets, or whatever you like. The host machine where the virtualized environment runs can be any machine with Linux (Ubuntu 22.04 in our case) with KVM virtualization enabled.

Prerequisites

First, verify that KVM virtualization is available on the host:

user@hostmachine:~/$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

Install the required packages:

sudo apt-get install qemu libvirt-daemon-system libvirt-clients ebtables dnsmasq-base
sudo apt-get install libxslt-dev libxml2-dev libvirt-dev zlib1g-dev ruby-dev
sudo apt-get install libguestfs-tools
sudo apt-get install build-essential

Install Vagrant according to the steps on the official installation page. Then install the Vagrant plugins:

vagrant plugin install vagrant-libvirt vagrant-hostmanager

If there is no SSH keypair in ~/.ssh, generate one. This keypair will be injected into the VMs, because cephadm — which we use for the Ceph deployment — needs SSH connectivity between the VMs, and this keypair is used for SSH authentication between nodes:

ssh-keygen

Deploy the cluster

Now we should be ready to start the virtual environment on the machine:

mkdir ceph-vagrant; cd ceph-vagrant
wget https://gist.githubusercontent.com/kmadac/171a5b84a6b64700f163c716f5028f90/raw/1cd844197c3b765571e77c58c98759db77db7a75/Vagrantfile

vagrant up

When vagrant up ends without any error, Ceph will be installed in the background for a couple more minutes. You can check the deployment progress by opening the Ceph shell on node0:

vagrant ssh ceph1-node0
vagrant@ceph1-node0:~$ sudo cephadm shell
root@ceph1-node0:/# ceph -W cephadm --watch-debug

Verify cluster health

At the end, you should get a healthy Ceph cluster with 3 MON daemons and 6 OSD daemons:

root@ceph1-node0:/# ceph -s
  cluster:
    id:     774c4454-7d1e-11ed-91a2-279e3b86d070
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph1-node0,ceph1-node1,ceph1-node2 (age 13m)
    mgr: ceph1-node0.yxrsrj(active, since 21m), standbys: ceph1-node1.oqrkhf
    osd: 6 osds: 6 up (since 12m), 6 in (since 13m)

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   33 MiB used, 60 GiB / 60 GiB avail
    pgs:     1 active+clean

Where to go from here

Now your cluster is up and running, and you can install additional services like CephFS or RGW, play with adding and removing nodes, and upgrade to the next version. By changing the CLUSTER_ID variable in the Vagrantfile and copying it to another directory, you can deploy a second cluster and try to set up replication (rbd-mirror, cephfs-mirror, RGW multizone configuration) between clusters. You are only constrained by the boundaries of your imagination.

When you are done with your tests, you can simply destroy the environment:

vagrant destroy -f

Where We Apply This

Reliable storage is foundational to the platforms we build and operate for clients in regulated, high-stakes environments — and a safe place to rehearse upgrades and configuration changes is part of running it well. Explore our infrastructure services or meet the team behind Grow2FIT.