K8s RPi

Configs for my Raspberry Pi Kubernetes Cluster (named "rstack") running k3s.

Glamour shots of the cluster:

This is the second iteration of my cluster setup (I wrote a blog post about my first iteration). Previously I was using full K8s with MetalLB for load balancing, now I'm using k3s with Traefik.

Standing up the Cluster

The ansible folder contains playbooks for provisioning and standing up the cluster, as well as for admin tasks such as bulk upgrading system packages.

Requirements / Prep steps

• Install ansible and ansible-playbook.
• Download and dd rasbian onto the Raspberry Pi SD cards.
• Mount the boot partition of each SD card and touch the file <bootpart>/ssh.
• Put SD cards into Raspberry Pis and boot them.
• Accept SSH signatures and enable password-less SSH by running ssh-copy-id pi@pi-address on each of the Pis (will help ansible).
• SSH into each pi and change the password for the pi user.
• Add static IP assignments to your router for each of the Pis (grab their MAC addresses while you are SSH'd in).
• Add each pi as a host in /etc/hosts (for ansible)
• Update ansible/hosts with each host name.
• Reboot the Pis so they pick up their static IPs. Double check the IPs have been leased properly by SSHing into each at their static IP/hostname.

Updating OS and Packages

To perform system updates (which you should to ensure security patches etc are installed), run:

ansible-playbook -i ansible/hosts ansible/update-all.yml

Installing k3s

Simply run the ansible playbook and wait:

ansible-playbook -i ansible/hosts ansible/provision-k3s.yml

Post Install

Setting up kubectl

The master node will have a kubectl config folder at /root/.kube. Copy this folder from the master node over to your computer somewhere (e.g. with rsync) so you can run kubectl commands from your computer.

To set this kubectl config as your default config, export the KUBECONFIG variable in your .bashrc file:

export KUBECONFIG=/path/to/.kube/config

Test kubectl to check everything is working (don't forget to source ~/.bashrc):

kubectl get nodes

NAME           STATUS   ROLES    AGE   VERSION
rstack-node0   Ready    master   91d   v1.15.4-k3s.1
rstack-node2   Ready    worker   91d   v1.15.4-k3s.1
rstack-node1   Ready    worker   91d   v1.15.4-k3s.1
rstack-node3   Ready    worker   91d   v1.15.4-k3s.1

Setting up Traefik (with LetsEncrypt certs working as well)

Traefik works great as an Ingress controller with k3s (I couldn't get ingress-nginx to work).

The LetsEncrypt setup uses DNS validation via Cloudflare, so your domain's DNS needs to hosted with Cloudflare (although other providers do work - see Traefik's Provider Docs).

Install Helm

Traefik is installed via helm, so we need to install helm first. Use the helper script:

bash ./install-helm.sh

Setup Cloudflare API keys

LetsEncrypt needs your Cloudflare API key (the 'Global' one, not an API 'token') to validate DNS challenges. Add it as a k8s secret:

kubectl -n kube-system create secret generic cloudflare-api-key \
  --from-literal=CLOUDFLARE_EMAIL=youremail@incloudflareaccount \
  --from-literal=CLOUDFLARE_API_KEY=cloudflareapikey

Setup Traefik config

Update the values in config/traefik/values.yml - notably acme.email to match your CloudFlare email.

You may also want to enable the dashboard with authentication, example yaml:

dashboard:
  enabled: true
  domain: customdomain.name
  auth:
    basic:
      customusername: password-generated-with-htpasswd

See Traefik's BasicAuth Docs for more info.

Install Traefik

Finally we can install Traefik:

helm install stable/traefik --name traefik --namespace kube-system --values config/traefik/values.yml

Test Traefik

There is a test nginx deployment/service/ingress at config/traefik/test-ingress.yml. Update the domains/host values and apply:

kubectl apply -f config/traefik/test-ingress.yml

Delete it with:

kubectl delete -f config/traefik/test-ingress.yml

Uninstall Traefik

In case you need to rollback the traefik install, run:

helm delete traefik && helm del --purge traefik
kubectl delete secret -n kube-system cloudflare-api-key
kubectl delete -f config/traefik/test-ingress.yml

Kubernetes Deployments/Configs

The config folder contains a number of K8s deployments/services.

• config/dashboard - (not in use) Kubernetes Dashboard deployment files for monitoring the Cluster.
• config/dnscrypt - DNSCrypt service for encrypted DNS (for home network).
• config/sensor-monitoring - Contains all the microservices for my IoT Home Monitoring setup, which monitors the temperature around my house using MQTT, Influxdb, and Grafana, and my computer stats using Prometheus.
• config/docker-cloudflare-ddns.yml - Dynamic DNS service if you do not have a static IP (uses same Cloudflare API key as traefik).
• config/hue-im-home.yml - Hue-Im-Home service to automatically turn off/on house lights when leaving for/coming home from work.

License

I hope this repo is of help to anyone else setting up k3s on a Raspberry Pi Cluster.

MIT - see LICENSE.md