# Veda The new setup of my homelab will be based on Kubernetes, which will prevent all of my services going down when I need to do physical maintenance of a host. ## Services ### Core - Ceph for all storage: cephfs, object storage and block storage - Nextcloud: file storage interface for the entire family - Jellyfin: Web based media streaming - Authentik: Central identification and authentication server - Nginx reverse proxy - ACME client: SSL certificate handling - ArgoCD: Revision control for all Kubernetes configuration - Homeassistant + Zigbee2mqtt - Prometheus - Grafana - Grafana Loki + FluentD - Cilium - Harbor: Container image storage ### Nice-to-have - Jellyseerr: Nice interface to request movies and series - Sonarr: Automated downloading and handling of series - Radarr: Automated downloading and handling of movies - Flaresolverr: Fetching data hidden behind captcha’s - Torrent client (qBittorrent): To download all the linux ISO’s - ExternalDNS - Paperless-ngx ### Look-into-later - Mastodon: federated social platform - Forgejo: Git platform. Maybe this should not be hosted on the cluster as it will depend on it. - CloudNativePG: K8s operator for PostgreSQL ## Installing ### Configuration ```bash export CLUSTER_NAME="veda" export API_ENDPOINT="https://192.168.0.1:6443" ``` ```bash talosctl gen secrets --output-file secrets.yaml ``` ```bash talosctl gen config \ --with-secrets secrets.yaml \ --output-types talosconfig \ --output talosconfig \ $CLUSTER_NAME \ $API_ENDPOINT ``` ```bash talosctl config merge ./talosconfig ``` Then correct the endpoint in the Talos client configuration: ```yaml # ~/.talos/config context: veda contexts: veda: endpoints: - 192.168.0.1 # (...) ``` For control plane nodes: ```bash talosctl gen config \ --output rendered/master1.yaml \ --output-types controlplane \ --with-secrets secrets.yaml \ --config-patch @nodes/master1.yaml \ --config-patch @patches/network.yaml \ --config-patch @patches/scheduling.yaml \ --config-patch @patches/discovery.yaml \ --config-patch @patches/diskselector.yaml \ --config-patch @patches/vip.yaml \ --config-patch @patches/metrics.yaml \ --config-patch @patches/hostpath.yaml \ $CLUSTER_NAME \ $API_ENDPOINT ``` For worker nodes: ```bash talosctl gen config \ --output rendered/worker1.yaml \ --output-types worker \ --with-secrets secrets.yaml \ --config-patch @nodes/worker1.yaml \ --config-patch @patches/network.yaml \ --config-patch @patches/scheduling.yaml \ --config-patch @patches/discovery.yaml \ --config-patch @patches/diskselector.yaml \ --config-patch @patches/metrics.yaml \ --config-patch @patches/hostpath.yaml \ $CLUSTER_NAME \ $API_ENDPOINT ``` ### Bootstrapping Apply the configuration to each node: ```bash talosctl apply-config --insecure --file rendered/master1.yaml --nodes 192.168.0.10 ``` Optionally, check the status. Point the Talos API endpoint directly to the node, since etcd, and thereby kube-vip, is not up. ```bash talosctl -n 192.168.0.10 -e 192.168.0.10 dashboard ``` To start the cluster, we need to bootstrap the etcd cluster. This only has to be done for a single node. ```bash talosctl -n 192.168.0.10 -e 192.168.0.10 bootstrap ``` Finally, retrieve the kubeconfig, it will merge with `~/.kube/config`, if it exists. ```bash talosctl -n 192.168.0.10 kubeconfig ``` Check nodes, note the NotReady status, since the Cilium CNI is not running yet: ```bash kubectl get nodes ``` Install the Gateway API: ```bash kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml ``` Install Cilium: ```bash bash scripts/cilium.sh ``` ## TODO - Remove secrets from config ## Misc ### Applying patches ```bash talosctl patch machineconfig -p @argocd.yaml -n 192.168.0.0 ``` ### Reset node ```bash talosctl reset --system-labels-to-wipe EPHEMERAL,STATE --reboot -n 192.168.0.0 ``` ### ArgoCD default login User: admin, password can be retrieved with (ignore the '%' at the end): ```bash kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ``` ### Ceph default login User: admin on [http://ceph.noxxos.nl](http://ceph.noxxos.nl) ```bash kubectl -n ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo ``` ### Wiping disks for Ceph Start a temporary pod on each node where the disks are: ```bash kubectl run -it --rm \ -n ceph \ --image quay.io/ceph/ceph:v19.2.2 \ --privileged \ --overrides='{"spec": { "nodeSelector": {"kubernetes.io/hostname": "master3"}}}' fix ``` Search for the correct disk with `blkid`, set `DISK=/dev/sdX`, then run (some of) the following commands: ```bash ceph-volume lvm zap $DISK --destroy wipefs -a $DISK # Zap the disk to a fresh, usable state (zap-all is important, b/c MBR has to be clean) sgdisk --zap-all $DISK # Wipe portions of the disk to remove more LVM metadata that may be present dd if=/dev/zero of="$DISK" bs=1K count=200 oflag=direct,dsync seek=0 # Clear at offset 0 dd if=/dev/zero of="$DISK" bs=1K count=200 oflag=direct,dsync seek=$((1 * 1024**2)) # Clear at offset 1GB dd if=/dev/zero of="$DISK" bs=1K count=200 oflag=direct,dsync seek=$((10 * 1024**2)) # Clear at offset 10GB dd if=/dev/zero of="$DISK" bs=1K count=200 oflag=direct,dsync seek=$((100 * 1024**2)) # Clear at offset 100GB dd if=/dev/zero of="$DISK" bs=1K count=200 oflag=direct,dsync seek=$((1000 * 1024**2)) # Clear at offset 1000GB # SSDs may be better cleaned with blkdiscard instead of dd blkdiscard $DISK # Inform the OS of partition table changes partprobe $DISK ``` ### Certificate lifetimes Talos Linux automatically manages and rotates server-side certificates for etcd, Kubernetes, and the Talos API. Note however that the kubelet needs to be restarted at least once a year in order for the certificates to be rotated. Any upgrade/reboot of the node will suffice for this effect. You can check the Kubernetes certificates with the command `talosctl get KubernetesDynamicCerts -o yaml` on the control plane. Client certificates (`talosconfig` and `kubeconfig`) are the user's responsibility. They are separate from the cluster's server-side certificates. Each time you download the kubeconfig file from a Talos Linux cluster, the client certificate is regenerated, giving you a kubeconfig which is valid for a year. The `talosconfig` file should be renewed at least once a year, before it expires. If the current `talosconfig` is still valid, renew it through a control plane node: ```bash cd talos talosctl -n 192.168.0.10 config new talosconfig --roles=os:admin talosctl config merge ./talosconfig talosctl config endpoint 192.168.0.1 ``` If the current `talosconfig` client certificate is already expired, recover it from the stored cluster secrets instead. Do not run `talosctl gen secrets` for an existing cluster. ```bash cd talos talosctl gen config veda https://192.168.0.1:6443 \ --with-secrets secrets.yaml \ --output-types talosconfig \ --output talosconfig \ --force talosctl config merge ./talosconfig talosctl config endpoint 192.168.0.1 ``` After renewing `talosconfig`, regenerate the Kubernetes client certificate as well: ```bash talosctl -n 192.168.0.10 kubeconfig ``` ### Ceph host networking For some reason the Ceph object gateway is not properly configured in the dashboard. [See this issue for similar symptoms](https://github.com/rook/rook/issues/12099)