I’m a really big fan and believer of Kubernetes. I’m already working with Kubernetes for about 2 years, and I always wanted to know how to operate a larger Kubernetes Cluster. Therefore, back in the year 2018 I bought the necessary equipment to build a large-scale Kubernetes Cluster with 28 (!) Raspberry Pi’s! Unfortunately, I broke at that time my leg, and after that I never continued to work on that cool project. But this changed in the last few weeks as you can see:

This is a full-blown Kubernetes Cluster running on top of 28 (!) Raspberry Pi 3 B+ devices. In this blog posting I want to give you an overview about the used hardware and how I setup the Kubernetes Cluster itself.

The Hardware

Running 28 Pi’s in a cluster sounds to be a great, cool idea. But as soon as you think in more detail about it, a lot of different questions arise:

• How do you handle the power management?
• What’s about the necessary networking infrastructure?
• How to tell it to my wife?

Let’s have a look on these questions one by one. The Raspberry Pi devices can be powered through a Micro USB cable or through an additional PoE Hat (Power over Ethernet). Because I wanted to keep the costs down, I have decided to just use Micro USB cables to power the individual Raspberry Pi devices. But this also means that you have to deal with a lot of cables, and that you need a few USB chargers for the Micro USB cables. If I would build that project again, I would use the approach with the PoE Hats because you have less cables to manage.

How do you manage the networking infrastructure? I didn’t wanted to buy a huge network switch with a lot of ports, because I also wanted my Raspberry Pi Cluster as small as possible (back in these days travelling around the world was still allowed…). I have decided to use four 8-port switches, and one 5-port switch to satisfy my networking requirements. 7 Raspberry PIs are connected to each 8-port switch (7 x 4 = 28), and each 8-port switch is finally connected to the 5-port switch, which is finally connected to my home network.

And that’s also the reason why my cluster is currently limited to 28 Raspberry PIs – I’m out of network ports 😉 The pink cable in the following picture is the uplink on the 5-port switch into my home network.

How to tell it your wife? Well, that’s a different story…

I have physically connected all the 28 Raspberry Pi devices through 3mm spacers and mounted everything on a 1U shelf base. One drawback of this approach is that I can’t directly access the SD card slots of the Raspberry Pi devices. Therefore, I have to remove them from the shelf base when I want to flash the SD cards with a different OS.

Here is the detailed BOM list that I have bought back in the year 2018 for this project:

• 28x Raspberry Pi 3 B+
• 28x 16 GB SanDisk Ultra SD Cards
• 28x Micro USB Cables
• 4x 8-port USB Chargers
• 28x 0.5-meter Ethernet cables
• 4x Netgear GS108GE 8-Port Network Switches
• 1x Netgear GS105GE 5-Port Network Switch
• 1x 1U Shelf Base

Installing and configuring the OS

By now you have already a great overview about the hardware setup itself. Let’s do now something useful with it. As I have already said, I’m a big believer in Kubernetes, and I’m doing a lot of work with it these days. Therefore, I wanted to install a Kubernetes Cluster on this hardware setup.

One problem that you are facing with Kubernetes is that you need some RAM and CPU resources for the individual nodes. But as you know, back in the year 2018 the Raspberry Pi 3 B+ was the latest model. And this one has only 1 GB of RAM, which is almost nothing compared the current generation of the Raspberry Pi 4 with up to 8 (!) GB of RAM. Therefore, a full-blown traditional Kubernetes Cluster doesn’t really work quite well, because your available RAM is a limiting factor.

One alternative here is a k3s Kubernetes installation, which is a lightweight Kubernetes distribution suitable for IoT and Edge computing. It is provided by Rancher and it is certified by the Cloud Native Computing Foundation (CNCF).

In the first step I had to flash 28 SD cards with the Raspberry Pi Light OS. I have used here the Raspberry Pi Imager which made this task quite easy, expect the fact that I had to do it 28 times…

In addition, I have also enabled the SSH server after the OS image was flashed, so that I can use the Raspberry PIs in completely headless scenario: no monitor, no keyboard, no mouse.

sudo touch /Volumes/boot/sh

After the SD cards were prepared, I have inserted them into the Raspberry PIs, and started up the first node – which will become the Kubernetes Master Node. One current limitation of the k3s Kubernetes distribution is you can only have 1 Master Node, so you have currently no high availability for the Control Plane components…

When you power on the Raspberry Pi Light OS, it will appear in your network with the hostname raspberrypi.home. Because the SSH server is already running, you can SSH into the PI with the user pi and the password raspberry.

In the next step I have assigned a static IP address to the Raspberry Pi. You have to make these changes in the file /etc/dhcpcd.conf:

To be able to install and use Docker as a Container platform (containerd would also work for k3s), you have to enable the Container features in the kernel. For that you have to edit the file /boot/cmdline.txt by adding the following settings:

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

And finally, I executed the configuration program raspi-config and made the following changes:

• Changing the password of the pi user
• Setting a host name
• Changing the time zone

And then I performed a reboot to apply these changes.

Setting up the Kubernetes Master Node

With all these changes in place we are now finally able to install and setup the Kubernetes Master Node. The Kubernetes Master Node is executed on the Raspberry Pi with the host name rpi1 and the IP address 192.168.100.101. First of all, we have to install Docker as the Container platform by executing the following command:

sudo apt install docker.io -y

After the installation completed, I just verified the Docker installation with the following commands:

sudo systemctl start docker
sudo systemctl enable docker
sudo systemctl status docker
sudo docker version

And now we are able to start the installation of the k3s Kubernetes Master Node by executing the following command:

curl -sfL https://get.k3s.io | sh -s - --docker

As you can see from that command line, you are specifying here Docker as your Container platform. And finally, your Kubernetes Master Node is up and running. You can verify it by running the following command:

sudo kubectl get nodes -o wide

Setting up the Kubernetes Worker Nodes

After we have installed our Kubernetes Master Node, it’s now time to install and configure the individual Worker Nodes – in my case 27… I did here the same OS preparations as with the Master Node. The only things that were changed are the host name and the IP address. In the next step we have again to install Docker and verify its installation:

sudo apt install docker.io -y
sudo systemctl start docker
sudo systemctl enable docker
sudo systemctl status docker
sudo docker version

And finally, we have to install Kubernetes and join the new Worker Node to the Master Node. To be able to perform this, your Worker Node needs a so-called Join Token, which can be retrieved on the already installed Master Node. You can retrieve this Join Token by executing the following command on the Master Node:

sudo cat /var/lib/rancher/k3s/server/node-token

With that Join Token and the IP address of the Kubernetes Master Node in your hand, you can now finally install k3s on the Worker Node and join it into the Kubernetes Cluster:

curl -sfL http://get.k3s.io | K3S_URL=https://192.168.100.101:6443 K3S_TOKEN=K10273317d60da3719066061e9efccb11760dac101b34a5100d02f344429262a949::server:d2dcf361f5b07fa7103d9e682842f64a sh -s - --docker

I had to perform these steps now on all remaining Raspberry Pi devices. And finally, I had a fully functional Kubernetes Cluster:

As I have already said previously, one (current) drawback of k3s is the fact that it only supports a single Master Node. So, this is your single point of failure. Be aware of that! Maybe this will change in a later version of k3s. But currently this doesn’t hinder me to use this Cluster to learn about operating Kubernetes in a large-scale environment.

When you look in more detail on the previous screenshot, you maybe asking why my cluster has “only” 27 Raspberry Pi devices? Well, the answer is quite easy: One Pi doesn’t work since I got it, so that 28th switch port is currently empty 😉

The following short video shows the startup sequence of the Raspberry Pi Kubernetes Cluster – look at all those fancy lights!

Summary

I have bought all these Raspberry Pi devices back in the year 2018, when the Raspberry Pi 3 B+ was the latest model. But now we have already a Raspberry Pi 4 available, which supports up to 8 GB RAM. This device would be an ideal candidate to run a “real”, full-blown Kubernetes Cluster. Therefore, I’m currently already thinking about buying 4 of these PIs to build an additional cluster – maybe in combination with a BitScope Cluster Blade. But please: don’t tell this my wife…

Thanks for your time,

-Klaus