How to prepare dev/test Ceph environment
When you are using Ceph in production, it is important to have an environment where you can test your upcoming upgrades, configuration changes, integration of new clusters, or any other significant changes without touching real production clusters. Such an environment can be built with the tool called Vagrant, which can very quickly build virtualized environment described in one relatively simple config file.
We are using Vagrant on Linux with libvirt and hostmanager plugins. Libvirt is a toolkit to manage Linux KVM VMs. Vagrant can also create virtualized networks to interconnect those Vms and storage devices, so you can have an almost identical copy of your production cluster if you need it.
Let‘s create 5 nodes Ceph cluster. The first 3 nodes will be dedicated for control node daemons, all nodes will also be OSD nodes (2 x 10 GB disks on each node by default), and one node will be a client node. Client nodes can be used for testing access to cluster services. Mapping rbd images, mounting CephFS filesystems, accessing RGW buckets, or whatever you like. The host machine where the virtualized environment run can be any machine with Linux (Ubuntu 22.04 in our case) with KVM virtualization enabled.
user@hostmachine:~/$ kvm-ok INFO: /dev/kvm exists KVM acceleration can be used
Install required packages:
sudo apt-get install qemu libvirt-daemon-system libvirt-clients ebtables dnsmasq-base sudo apt-get install libxslt-dev libxml2-dev libvirt-dev zlib1g-dev ruby-dev sudo apt-get install libguestfs-tools sudo apt-get install build-essential
Install vagrant according to the steps on the official installation page: https://developer.hashicorp.com/vagrant/downloads
Then we need to install vagrant plugins:
vagrant plugin install vagrant-libvirt vagrant-hostmanager
If there is no ssh keypair in ~/.ssh, generate one. This keypair will be injected into the Vms, because cephadm which we will use for Ceph deployment needs ssh connectivity between VMs and this keypair will be used for ssh authentication between nodes.
Now we should be prepared to start your virtual environment on the machine.
mkdir ceph-vagrant; cd ceph-vagrant wget https://gist.githubusercontent.com/kmadac/171a5b84a6b64700f163c716f5028f90/raw/1cd844197c3b765571e77c58c98759db77db7a75/Vagrantfile vagrant up
When vagrant up ends without any error, ceph will be installed in the background for a couple of more minutes. You can check deployment progress by accessing ceph shell on node0:
vagrant ssh vagrant ssh ceph1-node0 vagrant@ceph1-node0:~$ sudo cephadm shell root@ceph1-node0:/# ceph -W cephadm –watch-debug
at the end, you should get a healthy ceph cluster with 3 MON daemons and 6 OSD daemons:
root@ceph1-node0:/# ceph -s cluster: id: 774c4454-7d1e-11ed-91a2-279e3b86d070 health: HEALTH_OK services: mon: 3 daemons, quorum ceph1-node0,ceph1-node1,ceph1-node2 (age 13m) mgr: ceph1-node0.yxrsrj(active, since 21m), standbys: ceph1-node1.oqrkhf osd: 6 osds: 6 up (since 12m), 6 in (since 13m) data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 33 MiB used, 60 GiB / 60 GiB avail pgs: 1 active+clean
Now your cluster is up and running, and you can install additional services like CephFS or RGW, you can play with adding/removing nodes, and upgrading to the next version. By changing CLUSTER_ID variable in Vagrantfile and copying Vagrantfile to another directory, you can deploy a second cluster and try to setup replication (rbd-mirror, cephfs-mirror, RGW multizone configuration) between clusters. The boundaries of your imagination only constrain you.
When you are done with your tests, you can simply destroy the environment with
vagrant destroy -f
Grow2FIT Infrastructure Consultant
Kamil is a Senior Cloud / Infrastructure consultant with 20+ years of experience and strong know-how in designing, implementing, and administering private cloud solutions (primarily built on OpenSource solutions such as OpenStack). He has many years of experience with application development in Python and currently also with development in Go. Kamil has substantial know-how in SDS (Software-defined storage), SDN (Software-defined networking), Data storage (Ceph, NetApp), administration of Linux servers and operation of deployed solutions.
Kamil regularly contributes to OpenSource projects (OpenStack, Kuryr, Requests Lib – Python).
The entire Grow2FIT consulting team: Our Team