K8S on PIs, Choosing my Flavor

Choosing your own K8S Adventure

Note, here on I will be using K8S as the short form of Kubernetes. I expect to be writing K8S ... a lot! In the past, when K8S was young and new, I got excited. Having somehow convinced myself I understood how VMware was able to virtualize full running computers, on top of another computer, I was sold that K8S would take it one step further. And. convinced myself that docker was too limiting to be usable at scale, I dove face first off the high board into the empty pool of early Kubernetes adoption.

Well, not entirely. But convinced that a thinner layer across multiple hosts, able to leverage commodity servers to create a high level of abstraction between compute resources and run time code would be a solid evolution, I hopped on the GCP train and hand built my own Kubernetes cluster. And to be honest, it was simple, elegant, and rather trivial. This involved steps including rolling your our Certificate Authority, some shell scripting, and eventually a bit of keyboard smashing, and it was up. Glorious Kubernetes, running on GCP compute nodes. Then came making it useful, and well, I got bored and found something more interesting to do.

Then came KOPS. A heavily modified cloud formation script, coupled with a hefty bit of shell scripting, a few web ui clicks, and bam, a Kubernetes cluster! Mix in some workloads (was doing ethereum / hyperledger

fabric work, and it all worked quite well). Alas, that project died, and so did that K8S cluster.

But moving on, I decided to dig into the world of small scale K8S. K8S at the Edge some edgy marketing folks call it. But for me, it was K8S on Raspberry PIs I had sitting in a bin in my office. Why? Well, I had them in hand, and, the launch of ARM64 on AWS via their Graviton2 push meant growing adoption of ARM64 as a production compute solution, and the adoption (or at least multi-targeting of common K8S workloads) for ARM64.

But even then, there were 3 common contenders. In one corner, Ubuntu's MicroK8S, marketed as the lightest weight, easiest to deploy K8S clusters for edge compute. In the opposite corner, K3S, from the folks at Rancher. A light weight, ready to run, K8S solution with a growing community of contributors and one fellow who rights a lot of tools to support it, Alex Ellis. Finally, and often cited as a great solution, comes Minikube. There are many articles on the differences and benefits of each. And, so, we tried each.

I had used minikube as a quick and dirty way to get folks running K8S on MacOS. But long and short of it is, Minikube seems to be a great way to run multiple K8S nodes on a single host, using it to span 7 raspberry PIs .

Microk8s was trivial to get started with. Spanning 7 PIs was easy, so long as you are patient in when adding a new node to the cluster. Yes, I caused a cluster to hang, corrupt it's state database, and then had to recover by resetting each. Microk8s has a decent set of out of the box ready to enable capabilities, and a community of contributing addons, including many we hoped to work with, including ArgoCD. So I built that out. Adding a mix of InfluxDB, telegraf, and Grafana turned out fairly successful. But issues arose. First, and foremost, Calico, the CNI for MicroK8S kept crashing. The community eventually rallied around and found a working solution. But, well, I wasn't satisfied with how much of the actual configuration and control was taken out of my hands.

So I returned to K3S, and the Alex Ellis's K3Sup scripts. It started out compelling, but I immediately ran into oddities and headaches. That said, given the many charts, my desire for full control, and the want to use an external host for state management (covered in separate post), I believe K3S is the right direction to head.

Having committed to a decision, the next steps was to plan all the components and tech I'd use. As covered in the next part of this series.