Personal Cluster
We tried many options such as using Ceph when we designed our clusters and we end up with the following design. One key element in our selection was to remain independent of any commercial products. We strongly believe in free software. Another key element was the robustness.
We wanted to have several features available from day one:
- Hypervisor
- Software-defined storage & Scale out storage
- Disk Encryption
- Software-defined networking
- Data deduplication backup
- Telemetry
resulting in a hyper converged architecture.
Hardware requirements
We identified a typical hardware requirements which could be tune on demand.
- 3 regular PC with at least 3 hard drives / SSD and 3 network interfaces
- 3 network switches (L2): One 10 Gb and two 1 Gb
- a connection to internet
- A Rapsberry Pi or a NUC to run a tang server (To decrypt the disks at boot time without having to enter a password)
This procedure can be easily adapted to run a cluster of Raspberry Pi. We managed to run a cluster on 3 Raspberry Pi with limited network capabilities (for instance no routing / no firewall capabilites).
Components
We selected the following components to fulfill our requirements:
| Component | Feature |
|---|---|
| KVM / libvirt / virt-manager | Hypervisor |
| GlusterFS | Software-defined storage & Scale out storage |
| clevis / luks / tang | Disk Encryption |
| Open vSwitch / OPNsense | Software-defined networking |
| BorgBackup | Data deduplication backup |
| Prometeus / Grafana | Telemetry |
Network
Switch 1: 10 Gb
Switch 2 and 3: 1 Gb
You can run one or more OPNsense VM as router / firewall.
VLAN
You can create as many VLAN as you want using Open vSwitch and OPNsense. Your L2 switch should support VLAN tagging.
Storage
The 4 disks of each node will form 4 types of volumes:
- A disk for the Operating System
- A GlusterFS volume to store VM disks (gv0)
- A GlusterFS volume to store backup (gv1)
- A disk for barre metal VM disks such as VM running Ceph on Kubernetes (bmv0)
All disk are encrypted except the disk used by the Operating System.
Backup
BorgBackup lets you save a lot of space using data deduplication. You could run daily backup and keep a year of history of all VM disks on reasonable disk space.
GlusterFS allows you to snapshot an entire volume which can be helpfull backuping up VMs at once.