Distributed file storage with a Ceph cluster on Raspberry Pi

Single-board computers like Raspberry Pis offer a cost-effective way to build a multi-node cluster at home. This allows you to experiment with technologies like Kubernetes cluster, distributed storage systems, and serverless computing. In this guide, we'll explore building a 3-node Raspberry Pi 5 storage cluster using Ceph.

Ceph is an open-source software-defined storage platform that provides distributed file, block, and object storage functionalities. It allows you to create a fault-tolerant data storage system accessible over the TCP/IP protocol.

CephFS is a component of Ceph that provides a POSIX-compliant file system interface. Client systems can mount Ceph's RADOS object storage using CephFS, making it appear just like a standard Linux file system. This enables file sharing functionality similar to traditional solutions like NFS. Ceph Clients mount a CephFS filesystem as a kernel object or as a Filesystem in User Space (FUSE).

Ceph relies on several services to function:

• MON (Ceph Monitor - ceph-mon): Monitors are the brains of the Ceph cluster. They manage the overall health of the cluster, track the state of Object Storage Devices, and provide authentication for clients and daemons. For redundancy and high availability, at least three monitors are recommended.
• OSD (Object Storage Device - ceph-osd): OSDs store data and process client requests. Data within an OSD is divided and stored in blocks. Ceph OSD Daemons handle read, write, and replication operations on storage drives. At least three Ceph OSDs are normally required for redundancy and high availability. This can be a traditional hard disk (HDD) or a solid state disk (SSD).
• MDS (Metadata Server Daemon - ceph-mds): The MDS plays a crucial role in CephFS. It manages the metadata associated with the file system, allowing for efficient file operations. The Ceph Metadata Server is necessary to run Ceph File System clients.
• MGR (Manager Daemon - ceph-mgr): The MGR acts as a central monitoring and management service for the Ceph cluster. It is responsible for keeping track of runtime metrics and the current state of the Ceph cluster, including storage utilization, current performance metrics, and system load.

Here is a diagram of a Ceph cluster consisting of Raspberry Pi nodes:

My Ceph cluster will consist of three Raspberry Pi 5s, which will be connected in a private network using a 1Gbit switch. For storage, I am using three 256GB flash SSD drives. While solid-state drives are recommended for better performance, using USB 3.0 will limit the overall speed anyway. The Raspberry Pi 5 is based on the Debian 12 (bookworm) operating system, while the Raspberry Pi 4 is based on the Ubuntu 22.04 server OS.

My Ceph cluster looks like this:

In real-world deployments, Ceph clusters typically have many more OSDs than what's shown in the image. This is because Ceph relies on replication to protect data. Each object is replicated across multiple OSDs. If an OSD fails, its data can still be retrieved from the replicas, ensuring data redundancy and availability.

For more information on Ceph, you can refer to the official documentation: Intro to Ceph.

Prerequisites for Ceph Deployment

First we need to SSH to our deployment node, make sure you have setup IPs, network and /etc/hosts on all Pi nodes if you are not using local DNS and DHCP with static assignments.

Each Ceph node MUST be able to ping every other Ceph node in the cluster by its short host name. You can use the hostnamectl command to set appropriate hostnames on each node.

hostnamectl set-hostname {your hostname}

All Ceph nodes should be resolvable by hostname using the /etc/hosts file on each machine. Here's an example /etc/hosts configuration for your scenario:

192.168.0.100 deployment
192.168.0.106 ceph1
192.168.0.107 ceph2
192.168.0.108 ceph3

We also need to make sure the system clocks are not skewed so Ceph cluster can operate properly. It is recommended to sync the time with NTP server.

Since ceph-deploy will not prompt for a password, you must generate SSH keys on the deployment node and distribute the public key to each

ssh-copy-id <username>@node1 
ssh-copy-id <username>@node2 
ssh-copy-id <username>@node3

These commands will add the admin node's public key to the authorized_keys files on each Ceph node, allowing passwordless SSH access for deployment tasks.

Ceph Storage Cluster Deployment with ceph-deploy

This guide details deploying a Ceph storage cluster using the ceph-deploy tool. ceph-deploy simplifies the process by automating many configuration steps.

The Raspberry Pi OS named Bookworm has been released simultaneously with the Raspberry Pi 5. Consequently, it has been upgraded from Debian 11 "Bullseye" to Debian 12 "Bookworm." Since Ceph release 18.2.0 (Reef) supports Debian 12 Bookworm, we will be focusing on installing this specific version.

On the deployment node, which is my case is Raspberry Pi 4 boards install ceph-deploy using pip:

pip3 install git+https://github.com/ceph/ceph-deploy.git

Run the following command to create a new cluster named ceph with monitors on nodes ceph1, ceph2, and ceph3:

ceph-deploy new ceph1 ceph2 ceph3

The output will display detailed information about each step.

[ceph_deploy.conf][DEBUG ] found configuration file at: /home/raspi/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.1.0): /usr/local/bin/ceph-deploy new ceph1 ceph2 ceph3
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose  : False
[ceph_deploy.cli][INFO ] quiet  : False
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] ceph_conf  : None
[ceph_deploy.cli][INFO ] cluster  : ceph
[ceph_deploy.cli][INFO ] mon  : ['ceph1', 'ceph2', 'ceph3']
[ceph_deploy.cli][INFO ] ssh_copykey  : True
[ceph_deploy.cli][INFO ] fsid : None
[ceph_deploy.cli][INFO ] cluster_network  : None
[ceph_deploy.cli][INFO ] public_network : None
[ceph_deploy.cli][INFO ] cd_conf  : <ceph_deploy.conf.cephdeploy.Conf object at 0xffff9315e110>
[ceph_deploy.cli][INFO ] default_release  : False
[ceph_deploy.cli][INFO ] func : <function new at 0xffff93481cf0>
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[ceph1][DEBUG ] connected to host: head
[ceph1][INFO ] Running command: ssh -CT -o BatchMode=yes ceph1 true
[ceph1][DEBUG ] connection detected need for sudo
[ceph1][DEBUG ] connected to host: ceph1
[ceph1][INFO ] Running command: sudo /bin/ip link show
[ceph1][INFO ] Running command: sudo /bin/ip addr show
[ceph1][DEBUG ] IP addresses found: ['192.168.0.106']
[ceph_deploy.new][DEBUG ] Resolving host ceph1
[ceph_deploy.new][DEBUG ] Monitor ceph1 at 192.168.0.106
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[ceph2][DEBUG ] connected to host: head
[ceph2][INFO ] Running command: ssh -CT -o BatchMode=yes ceph2 true
[ceph2][DEBUG ] connection detected need for sudo
[ceph2][DEBUG ] connected to host: ceph2
[ceph2][INFO ] Running command: sudo /bin/ip link show
[ceph2][INFO ] Running command: sudo /bin/ip addr show
[ceph2][DEBUG ] IP addresses found: ['10.122.116.1', '192.168.0.107']
[ceph_deploy.new][DEBUG ] Resolving host ceph2
[ceph_deploy.new][DEBUG ] Monitor ceph2 at 192.168.0.107
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[ceph3][DEBUG ] connected to host: head
[ceph3][INFO ] Running command: ssh -CT -o BatchMode=yes ceph3 true
[ceph3][DEBUG ] connection detected need for sudo
[ceph3][DEBUG ] connected to host: ceph3
[ceph3][INFO ] Running command: sudo /bin/ip link show
[ceph3][INFO ] Running command: sudo /bin/ip addr show
[ceph3][DEBUG ] IP addresses found: ['192.168.0.108']
[ceph_deploy.new][DEBUG ] Resolving host ceph3
[ceph_deploy.new][DEBUG ] Monitor ceph3 at 192.168.0.108
[ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph1', 'ceph2', 'ceph3']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.0.106', '192.168.0.107', '192.168.0.108']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...

This command performs several actions:

• Establishes passwordless SSH between the deployment host and Ceph nodes.
• Creates a Ceph cluster named ceph with initial monitors on ceph1, ceph2, and ceph3.
• The command above will generate a ceph.conf file which looks like this:

fsid = 3218618a-ce05-4277-9a0d-325063ce54cc
mon_initial_members = ceph1, ceph2, ceph3
mon_host = 192.168.0.106,192.168.0.107,192.168.0.108
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

Each Ceph cluster has it own unique fsid identity. Ceph provides two authentication modes: none (anyone can access data without authentication) or cephx (key based authentication).

On all Ceph nodes install the necessary Ceph packages using your distribution's package manager.

sudo apt-get install python3-ceph-argparse=16.2.11+ds-2 python3-ceph-common=16.2.11+ds-2 ceph-mgr-modules-core=16.2.11+ds-2 ceph-mgr=16.2.11+ds-2 python3-cephfs=16.2.11+ds-2

Install Ceph on all nodes using ceph-deploy install. Specify the desired Ceph release (e.g., reef) after the --release option

ceph-deploy install --release reef ceph1 ceph2 ceph3

Use ceph-deploy to create the initial monitors on the designated nodes:

ceph-deploy mon create-initial

Copy the generated Ceph configuration file from the deployment node to all Ceph nodes:

ceph-deploy admin ceph1 ceph2 ceph3

Now let’s verify the first cluster status by SSH into a Ceph node and run the command below to check the cluster health.

sudo ceph health

We can also check the cluster status using below command:

sudo ceph -s

This output will show the status of monitors, managers, OSDs, and other cluster components.

cluster:
    id:     8b7cb3f3-f4b0-4d17-8d08-aaaf0b5030b2
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim
 
  services:
    mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 44s)
    mgr: no daemons active
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

If we get HEALTH_OK health, it means the cluster was initialize successfully. Let’s go back to the deployment node and install the Manager daemon from using the following command

ceph-deploy mgr create ceph1

The Manager daemon is not strictly necessary for basic storage functionality but can provide additional features like monitoring and orchestration.

Important: Ensure the target devices are not in use and contain no critical data.

Use dd to wipe any existing data on the target devices (e.g., /dev/sda1):

dd bs=1M count=1 </dev/zero >/dev/sda1

Confirm the device type using file

sudo file -s /dev/sda1

The output should indicate data.

sudo file -s /dev/sda1
/dev/sda1: data

Run ceph-deploy osd create on the deployment node to create OSDs on the specified devices on each Ceph node (e.g. /dev/sda1)

ceph-deploy osd create --data /dev/sda1 ceph1
ceph-deploy osd create --data /dev/sda1 ceph2
ceph-deploy osd create --data /dev/sda1 ceph3

Replace /dev/sda1 with the actual device on each node

You should see a message indicating each host is ready for OSD use.

[ceph_deploy.osd][DEBUG ] Host ceph1 is now ready for osd use.
[ceph_deploy.osd][DEBUG ] Host ceph2 is now ready for osd use.
[ceph_deploy.osd][DEBUG ] Host ceph3 is now ready for osd use.

Check the cluster health using ceph status:

cluster:
    id:     3218618a-ce05-4277-9a0d-325063ce54cc
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 26m)
    mgr: ceph2(active, since 26m), standbys: ceph1
    mds: 1/1 daemons up, 2 standby
    osd: 3 osds: 3 up (since 26m), 3 in (since 2d)
 
  data:
    volumes: 1/1 healthy
    pools:   3 pools, 65 pgs
    objects: 22 objects, 6.8 KiB
    usage:   35 MiB used, 715 GiB / 715 GiB avail
    pgs:     65 active+clean

Our cluster seems to be healthy and all three monitors and osds are listed under services.

Install the Ceph Manager Dashboard package on the node where the Manager is running:

sudo apt install ceph-mgr-dashboard=16.2.11+ds-2

Replace the version number (16.2.11+ds-2) if needed.

Enable the Dashboard module

sudo ceph mgr module enable dashboard

Create a self-signed certificate for authentication:

sudo ceph dashboard create-self-signed-cert

The output should confirm "Self-signed certificate created."

Create a user for dashboard access:

sudo ceph dashboard ac-user-create admin -i password administrator

View available Ceph Manager services:

sudo ceph mgr services

The output should display the Dashboard URL like:

{
"dashboard": "https://YOUR_IP_ADDRESS:8443/"
}

After the above configuration is completed, enter https://YOUR_IP_ADDRESS:8443/in the browser to enter the user name password to access the Ceph Dashboard and manage your Ceph storage cluster.

The Ceph dashboard cluster view provides an overview of the health of your Ceph cluster by displaying the various Ceph daemons, their status, and their resource allocation.

The hosts section of dashboard displays an overview of the Ceph cluster's member hosts. This view would typically display information such as the hostname and current role of each host within the Ceph environment, including whether they serve as OSDы or other roles like metadata servers.

OSDs are responsible for storing the actual data objects within the Ceph cluster. These devices could be physical disks or other storage devices connected to the hosts that serve as OSDs.

Then, we need to configure the Metadata Server (MDS) for the Ceph Filesystem. We can use ceph-deploy from the deployment host to deploy and configure MDS on ceph1, ceph2, and ceph3 nodes:

ceph-deploy --overwrite-conf mds create ceph1 ceph2 ceph3

The screenshot from dashboard highlighting the presence of three Metadata Servers. Metadata servers are responsible for storing the metadata about the data stored on the OSDs, which is necessary for clients to access and manage their data within the cluster.

Next, we will create two RADOS pools: a data pool and a metadata pool for the Ceph Filesystem, using the following commands:

sudo ceph osd pool create cephfs_data 8
sudo ceph osd pool create cephfs_metadata 8

8 argument specifies the number of Placement Groups (PGs) for the pool. Here, a value of 8 is assigned, but the optimal number of PGs can vary depending on your specific cluster setup and workload.

Expected Output:

pool 'cephfs_data' created
pool 'cephfs_metadata' created

The screenshot below displays the Ceph storage pools. It highlights two newly created pools

After creating the required pools, we will create the Ceph Filesystem with the following command:

sudo ceph fs new cephfs cephfs_metadata cephfs_data

The expected output for this command should be:

new fs with metadata pool 3 and data pool 2

Verify the status of CephFS and MDS using the following commands:

ceph mds stat
ceph fs ls

The expected output should be:

cephfs:1 {0=ceph3=up:active} 2 up:standby
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]

Make sure the MDS is up and active before you proceed.

Then create a CephFS secret file. We would run the following command to create a secret file from the key to be used for authentication to mount on the client.

sudo ceph-authtool -p ./ceph.client.admin.keyring > ceph.key

You can also create a user keyring that we can use in both solutions for authorization and authentication as we have cephx enabled using below command.

sudo ceph auth get-or-create client.user mon 'allow r' mds 'allow r, allow rw path=/home/cephfs' osd 'allow rw pool=cephfs_data' -o /etc/ceph/ceph.client.user.keyring

Finally, we’ll copy the secretfile and ceph.conf to the client machine.

CephFS offers two ways of mounting the filesystem – kernel or FUSE client

Mount CephFS with the Kernel Driver

To mount CephFS using the kernel driver, you need to have the Ceph kernel client module installed on your system. Use the following command to mount CephFS using the kernel driver:

sudo mount -t ceph <MONITOR_IP>:/<PATH> /mnt/cephfs -o name=<USERNAME>,secret=<SECRET_KEY>

In my case I run

mount -t ceph ceph1,ceph2,ceph3:/ /mnt/cephfs -o name=admin,secretfile=/etc/ceph/ceph.key,noatime

Mounting successful. To verify the mount, use the following command:

df -h

The output should be:

Filesystem  Size Used Avail Use% Mounted on
tmpfs 380M 3.1M 376M  1% /run
/dev/mmcblk0p2  118G  15G  99G 13% /
tmpfs 1.9G  0 1.9G  0% /dev/shm
tmpfs 5.0M  0 5.0M  0% /run/lock
/dev/mmcblk0p1  253M 137M 116M 55% /boot/firmware
tmpfs 379M 4.0K 379M  1% /run/user/1000
192.168.0.106,192.168.0.107,192.168.0.108:/ 227G  0 227G  0% /mnt/cephfs

To unmount the Ceph filesystem, use the following command:

umount /mnt/cephfs

This command unmounts the Ceph filesystem that was mounted at /mnt/cephfs

To automatically mount the Ceph filesystem at boot, edit the fstab file:

sudo nano /etc/fstab

Paste the following configuration at the end of the file:

# CephFS Mount
ceph1,ceph2,ceph3:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/ceph.key,noatime 0 0

Save the file and exit nano editor

Mount all disks from the fstab with the mount -a command, then list all available disks:

sudo mount -a

Finally, reboot the system.

After rebooting, use df -hT to confirm that the CephFS is still mounted at the designated location.

Starting and stopping all daemons

To start all Ceph services on a particular node,

systemctl start ceph.target

To stop all Ceph services on one particular node,

systemctl stop ceph\*.service ceph\*.target

Conclusion and further improvements

The deployment tool ceph-deploy has been deprecated in favor of cephadm for several Ceph releases now. Essentially, cephadm is the modern replacement for ceph-deploy. In the next tutorial, I will demonstrate how to expand a Ceph storage cluster and use the cephadm utility for deployment and maintenance.

Thank you for reading! If you'd like to learn more about Ceph clusters, I highly recommend checking out the references listed at the end of this post.

References

• Deploy Ceph storage cluster on Ubuntu server
• Use ceph-deploy to complete the quick installation of ceph distributed storage.
• KB450404 – Creating Client Keyrings for CephFS
• How to Deploy a Ceph Storage to Bare Virtual Machines
• Auto mount CephFS with autofs
• Deploy Ceph Using Cephadm
• How to Mount CephFS on CentOS 7
• CephFS Quick Start