Kubernetes: Planning for Success

Inspiration

Boredom, stuck inside during a pandemic, frustration with battling outside of my control issues with EKS, GKE, and even KOPS on EC2 instance, I started this project with the goal of getting immersed in the world of Kubernetes.

Kubernetes is not a monolith. It's an ecosystem, with a growing number of consumers, products, open source projects, and definitions of Cloud Native that feel slippery and complicated to start, and get more so as you get into it.

But, like Orpheus, marching onward with a song in our heart, and the belief that a simple tune can solve the ecosystems ills, we began our journey.

Aspiration

Delivery a unique, efficient, reliable, manageable, and esthetically pleasing Kubernetes cluster while not sacrificing on observability, CI/CD, and GitOps.

Execution

Being someone who has a long history of being exception at seeing and end state and working backwards, I began with a list. A structure of what I wanted to build, and the tooling and dependencies for each to help ensure and align on how I'd get it working. Our intent is to use this list, and blog along, as we work through each step of the way. Documenting references we relied upon, our experience in getting the work done, and the end result.

1. Giving Back - Setting up a blog to document this journey (and why, at the end of the day, all options suck, but some less than others).

2. Establishing the rules and restrictions - Setting up the base standards and working practices for a small team

3. Hard and Infrastructure - The building a POE powered, Immersion Cooled, Raspberry PI cluster, with RGB LED lighting, as well as the external control host, and the hardware management (via PI3).

4. Setting Up the Hosts - OS Flashing, setting up credentials, and other root activities to enable the future state.

5. Building the Control Host - First, why we chose to not host state on the PIs, why not to run observability, CI/CD, and other tooling on the cluster.

   1. Choosing and Building our relational Datastore - Postgresql setup, PGadmin setup, Redash, and creating proper maintenance plans

   2. OpenXPKI and the need for unified consistent cert management

   3. PowerDNS and the joy of delegation, subdomaining, and the critical role of DNS in infrastructure

   4. OpenIAM, Auth0, OIDC, and the sheer nightmare of self-hosted, small scale, federate authentication.

   5. NFS - the old school network file share (and why we maybe went with Min.io)

6. Automating workload delivery - GitOps, CI/CD, and IaaC in a modern K8S centric environment

   1. Self-Hosting GitLab - Source code and container repos are the foundation of any engineering team.

   2. Ansible and Semaphore - Automating host configuration management with GitOps style GitLab integration, and a WebUI.

   3. Container building with GitOps and Kaniko

   4. ArgoCD install and integration to GitLab

   5. Taking a pause and building our K8S Cluster

7. K3S on Raspberry PI via Ansible

   1. ArgoCD and GitLab integration to K3S

   2. Key-subsystems management via GitOps - MetalLB, Traefik, CertManager, CoreDNS, and Kube-Dash

   3. Custom container builds with multi-arch targetting in layers, like an onion, or a parfait

   4. Kustomize, Helm, and Kubes Manifests - GitOps with custom deployment definitions

8. Modern Observability for PIs and K8S

   1. InfluxDB - time series database on our control host

   2. Telegraf and PI - capturing core host metrics and pushing to Influx

   3. Grafana - the magic of WYSIWYG dashboarding

   4. Prometheus and Thanos - Scraping all the things cause some of it may be useful, the joy of K8S monitoring

   5. Jaeger - No, not the anise flavored liquor

9. Getting to the real project

   1. Basics of a service mesh, ISTIO, and modern microservice architecture

   2. A brief history of why I hate microservice architectures as a concept and why readily available compute resources means I (maybe/am) wrong

   3. Whatever comes next