kind create cluster
flux bootstrap git --private-key-file=C:/Users/Joe/.ssh/gitea --url ssh://git@gitea.home.joemonk.co.uk:2222/joe/gitops.git --branch main --path=clusters/kind

Kairos

Grab the latest image from https://github.com/kairos-io/kairos/releases, the image should have the format kairos-debian-bookworm-standard-amd64-generic-v3.1.1-k3sv1.30.2+k3s1. The main things we're looking for are the latest debian, standard, amd64, then the versions of kairos (v3.1.1) and k3s (1.30.2).
- Update the image at https://gitea.home.joemonk.co.uk/joe/kairos-custom to the latest kairos image and build it if additional tooling is needed in the image
Burn to usb
- Rufus can struggle with the image, Ventoy worked perfectly using the live image launch
Boot from usb, live install and go to the config webui
- If doing the firebat and it doesn't boot into bios or the drive, in grub press c then type fwsetup to reboot into bios
Add the public keys to the config (from ~/.ssh - ssh-keygen -t ed25519 -C "joemonk@hotmail.co.uk")
Update the image in the kairos-config to reflect the image being used, as well as any ssh keys or additional changes needed
Put the kairos-config in, check shutdown and let it install
Remove the usb & start the machine, wait for full boot
ssh in with using the specific private key added in the config (i.e. from ~/.ssh - ssh -i ./kairos kairos@192.168.1.101 or add the following to ~/.ssh/config to just use ssh 192.168.1.101)

Host 192.168.1.101
    HostName 192.168.1.101
    User kairos
    IdentityFile ~/.ssh/kairos

Adding additional packages

Go to https://gitea.home.joemonk.co.uk/joe/kairos-custom and add the new packages to the dockerfile
This image will be built when pushed
Follow the steps to upgrade/reinstall with the new image in the config - or just upgrade the image to the new image

Upgrading

SSH into the server and run sudo kairos-agent upgrade --source oci:gitea.home.joemonk.co.uk/joe/kairos-custom:3, restart and confirm all is good then run sudo kairos-agent upgrade --recovery --source oci:gitea.home.joemonk.co.uk/joe/kairos-custom:3 to update the recovery image too.

Reset

A full reset is a bit of a pain, as as far as I can tell, the "normal" reset keeps the current k8s state and data, which is probably not why we're after resetting.

⚠️ This will delete everything.

First of all, ensure the system-update-controller is installed on kairos (run from server/pc with kairos context) - kubectl apply -k github.com/rancher/system-upgrade-controller
You can then modify the kairos-reset.yaml to include the latest images, and kairos-config.yaml, and apply it with cat reset.yaml | kubectl apply -f -
This should then take a few minutes to reset the machine and reboot, meaning we can ssh in, grab the kubeconfig and re-bootstrap flux to reinstall everything.

Kubectl

SSH into the server, and grab the kubeconfig with sudo cat /etc/rancher/k3s/k3s.yaml. Drop the user and cluster into your config and create a context to have that user and cluster

age & sops

I use sops with age to encrypt keys etc in git.
Before pushing encrypted keys up, we'll need to re-encrypt them with a new key.
From a shell with sops and age installed, and an already known key under $HOME/.config/sops/age/keys.txt (or %AppData%\sops\age\keys.txt), create a new key in this repo age-keygen -o age.agekey.

Add that new public key to the .sops.yaml

In fish you can updatekeys in every secret (can just change to the bash equivalent if using bash)
for file in $(grep --include="*.yaml" -lr "sops:"); sops updatekeys -y $file; end

Then push the secret key to the cluster with: (This will need to be done after the initial flux bootstrap to get the namespace created, bt the keys should be updated first)

cat age.agekey |
kubectl create secret generic sops-age \
--namespace=apps \
--from-file=age.agekey=/dev/stdin

Delete age.agekey after sending it to the cluster.

Using sops

Encrypting

After creating a new secret, run sops -e -i ./path/to/secret.yaml.

Editing

You can install the @signageos/vscode-sops extension in vscode to automatically decrypt, edit and re-encrypt a secret.
Or use sops edit file.yaml

Flux CD

Install flux and everything in this repo with the following:

flux bootstrap git --private-key-file=/config/.ssh/gitea --url ssh://git@gitea.home.joemonk.co.uk:2222/joe/gitops.git --branch main --path=clusters/kairos

Reconcile

If changes aren't being brought across correctly (maybe the sops key has been updated after the flux bootstrap...), force a reconciliation with flux reconcile source git flux-system
TODO - I don't actually know if this works, or it's just a timed thing that coincides

DNS

We need to point a dns server to the server so we can access things via hostname rather than needing complex routing.

In OPNSense

Make sure Services > UnboundDNS is active and working
In overrides, add the host as *, domain as k3s and value as the ip address of the server

You should be able to access http://traefik.k3s/dashboard#/ (at the time of writing, looking to route this properly)

Grafana

Grab the admin user password from the monitoring-grafana secret