Home-Lab Refresh: Kubernetes Cluster Installation

January 2022

Following setting up PowerDNS for my Homelab Refresh, the next step was to create a Kubernetes cluster for running containers. While I could run containers directly on hosts using Docker, a Kubernetes cluster will allow me to use a gitops workflow to provision containers. This is preferable to alternatives such as deploying containers via Ansible and Terraform for the following reasons.

  1. Better automation. A gitops workflow will deploy changes to containers once a change is made to the git repo compared to making the change and then manually running Ansible or Terraform commands.
  2. Encrypted secrets can be decrypted and used for deploying changes, which provides secure version control for secrets.
  3. Git is the source of truth for what is running in the cluster.
  4. Related to the last point, it is more reliable to revert a change if something breaks.
  5. Confidence that I can take a backup of database data, blow away the cluster, create a new cluster, provision the containers from git, import the database data, and have no divergences from the state the cluster was in prior to it being blown away.

This is also a good chance to get some practical experience with Kubernetes, as learning new technology is the point of running a homelab after all.

Things to note with the cluster is that we are using Corosync and Pacemaker to provide a highly available floating IP that is used as the API endpoint. Additionally the PKI certificates are generated external to the cluster so that they can be distributed using Ansible without needing to provide SSH connectivity between hosts. Other then this, kubeadm is used to create the cluster and join the nodes.

Setting this up has a few steps:

  1. Terraform
  2. PKI
  3. Ansible
  4. Kubernetes

This can be condensed to simply running the following commands.

cd terraform/infrastructure/kubernetes
terraform1.1 apply

cd ../../../k8s-pki
./pki-gen all

cd ../ansible
ansible-playbook -i production k8s-all.yml

Terraform

The first step for deploying the Kubernetes servers is to provision them using Terraform. I’ve published the Terraform configuration that I’ve used for deploying these servers.

Variables

terraform/infrastructure/kubernetes/terraform.tfvars

hypervisor_hosts = {
  "kvm1" = {
    "ip"   = "10.1.1.21",
    "user" = "root",
  },
  "kvm2" = {
    "ip"   = "10.1.1.22",
    "user" = "root",
  },
  "kvm3" = {
    "ip"   = "10.1.1.23",
    "user" = "root",
  },
}

virtual_machines = {
  "k8s-controller-01" = {
    "ip"   = "10.1.1.41",
    "os"   = "debian_10"
  },
  "k8s-controller-02" = {
    "ip"   = "10.1.1.42",
    "os"   = "debian_10"
  },
  "k8s-controller-03" = {
    "ip"   = "10.1.1.43",
    "os"   = "debian_10"
  },
  "k8s-worker-01" = {
    "ip"   = "10.1.1.51",
    "os"   = "debian_10"
  },
  "k8s-worker-02" = {
    "ip"   = "10.1.1.52",
    "os"   = "debian_10"
  },
  "k8s-worker-03" = {
    "ip"   = "10.1.1.53",
    "os"   = "debian_10"
  },
}

domain = "lab.alexgardner.id.au"

host_admin_users = {
  "adminuser" = "ssh-rsa AAAAB[...truncated...]NZe19",
}

network_gateway_ip     = "10.1.1.1"
network_nameserver_ips = "10.1.1.31, 10.1.1.32, 10.1.1.33"

Commands

cd terraform/infrastructure/kubernetes
terraform1.1 apply

PKI

For this cluster I’m using external PKI certificates to prevent the Certificate Authority Private Keys needing to be stored on the nodes themselves. To make this easier, I wrote a quick bash script to generate all required SSL certificates, copy them into the relevant Ansible file directories, and then encrypt them to new files so they can be stored in git.

With this, generating the PKI certificates used by Kubernetes is as easy as running a single command. However, this is not idempotent so it will generate and copy new certificates each time it is run.

Commands

cd k8s-pki
./pki-gen all

Ansible

The next step is to run Ansible on all the nodes that are part of this cluster, including both controller and worker nodes. I’ve published the Ansible configuration that was used to do this. Using this Ansible playbook, the Kubernetes cluster can be set up in one single playbook run.

Variables

ansible/group_vars/all.yml

---
domain: lab.alexgardner.id.au
email: alex+homelab@alexgardner.id.au

nameservers:
  - '10.1.1.31'
  - '10.1.1.32'
  - '10.1.1.33'
  - '10.1.1.1'
network_subnets:
  - '10.1.1.0/24'
  - '10.1.2.0/24'
  - '10.1.3.0/24'

firewall_servers_subnet: 10.1.1.0/24
firewall_wireless_subnet: 10.1.2.0/24
firewall_clients_subnet: 10.1.3.0/24

timezone: Australia/Sydney

admin_users:
  - adminuser

ansible/group_vars/k8s_controllers.yml

---
kubernetes_clients_hosts_list:
  - '10.1.3.100'
kubernetes_cluster_name: kubernetes
kubernetes_cluster_fqdn: k8s-controller.{{ domain }}
kubernetes_pod_network_cidr: 10.10.0.0/16
kubernetes_secrets_key_aescbc: 
#checkov:skip=CKV_SECRET_6:Unencrypted secrets are git-ignored

pacemaker_hosts_list: "{{ groups['k8s_controllers'] | map('extract',hostvars,'ansible_host') | list }}"
pacemaker_hosts: "{{ groups['k8s_controllers'] }}"
pacemaker_primary_host: "{{ groups['k8s_controllers'][0] }}"
pacemaker_cluster_name: k8s-controller
pacemaker_cluster_ip: '10.1.1.40'

Commands

cd ansible
ansible-playbook -i production k8s-all.yml

Kubernetes

There is now a Kubernetes cluster with 3 controller nodes and 3 worker nodes. This is confirmed by going onto one of the controller nodes and running kubectl get nodes. Note that the status is showing as NotReady, this is because no CNI provider has been set up yet to provide pod connectivity.

adminuser@k8s-controller-02:~$ sudo kubectl get nodes
NAME                STATUS     ROLES                  AGE     VERSION
k8s-controller-01   NotReady   control-plane,master   2m21s   v1.23.2
k8s-controller-02   NotReady   control-plane,master   70s     v1.23.2
k8s-controller-03   NotReady   control-plane,master   51s     v1.23.2
k8s-worker-01       NotReady   <none>                 17s     v1.23.2
k8s-worker-02       NotReady   <none>                 17s     v1.23.2
k8s-worker-03       NotReady   <none>                 17s     v1.23.2

This is ok though, as the cluster itself is up and the API can take commands, leading to the next step which is to set up the gitops workflow so that Kubernetes can be configured via git.

[Edit]: This was not actually ok and was a rookie mistake/assumption on my part. It was easy enough to fix though, which was the first thing I did while setting up the Gitops configuration.

Next up: Kubernetes Gitops

Back to article list