# Veda The new setup of my homelab will be based on Kubernetes, which will prevent all of my services going down when I need to do physical maintenance of a host. ## Services ### Core - Ceph for all storage: cephfs, object storage and block storage - Nextcloud: file storage interface for the entire family - Jellyfin: Web based media streaming - Authentik: Central identification and authentication server - Nginx reverse proxy - ACME client: SSL certificate handling - ArgoCD: Revision control for all Kubernetes configuration - Homeassistant + Zigbee2mqtt - Prometheus - Grafana - Grafana Loki + FluentD - Cilium - Harbor: Container image storage ### Nice-to-have - Jellyseerr: Nice interface to request movies and series - Sonarr: Automated downloading and handling of series - Radarr: Automated downloading and handling of movies - Flaresolverr: Fetching data hidden behind captcha’s - Torrent client (qBittorrent): To download all the linux ISO’s - ExternalDNS - Paperless-ngx ### Look-into-later - Mastodon: federated social platform - Forgejo: Git platform. Maybe this should not be hosted on the cluster as it will depend on it. - CloudNativePG: K8s operator for PostgreSQL ## Installing ### Configuration ```bash export CLUSTER_NAME="veda" export API_ENDPOINT="https://192.168.0.1:6443" ``` ```bash talosctl gen secrets --output-file secrets.yaml ``` ```bash talosctl gen config \ --with-secrets secrets.yaml \ --output-types talosconfig \ --output talosconfig \ $CLUSTER_NAME \ $API_ENDPOINT ``` ```bash talosctl config merge ./talosconfig ``` Then correct the endpoint in the Talos client configuration: ```yaml # ~/.talos/config context: veda contexts: veda: endpoints: - 192.168.0.1 # (...) ``` For controlplane nodes: ```bash talosctl gen config \ --output rendered/master1.yaml \ --output-types controlplane \ --with-secrets secrets.yaml \ --config-patch @nodes/master1.yaml \ --config-patch @patches/argocd.yaml \ --config-patch @patches/cilium.yaml \ --config-patch @patches/scheduling.yaml \ --config-patch @patches/discovery.yaml \ --config-patch @patches/diskselector.yaml \ --config-patch @patches/vip.yaml \ $CLUSTER_NAME \ $API_ENDPOINT ``` For worker nodes: ```bash talosctl gen config \ --output rendered/worker1.yaml \ --output-types worker \ --with-secrets secrets.yaml \ --config-patch @nodes/worker1.yaml \ --config-patch @patches/argocd.yaml \ --config-patch @patches/cilium.yaml \ --config-patch @patches/scheduling.yaml \ --config-patch @patches/discovery.yaml \ --config-patch @patches/diskselector.yaml \ $CLUSTER_NAME \ $API_ENDPOINT ``` ### Bootstrapping Apply the configuration to each node: ```bash talosctl apply-config --insecure --file rendered/master1.yaml --nodes 192.168.0.10 ``` Optionally, check the status. Point the Talos API endpoint directly to the node, since etcd, and thereby kube-vip, is not up. ```bash talosctl -n 192.168.0.10 -e 192.168.0.10 dashboard ``` To start the cluster, we need to bootstrap the etcd cluster. This only has to be done for a single node. ```bash talosctl -n 192.168.0.10 -e 192.168.0.10 bootstrap ``` Finally, retrieve the kubeconfig, it will merge with `~/.kube/config`, if it exists. ```bash talosctl -n 192.168.0.10 kubeconfig ``` Check nodes: ```bash kubectl get nodes ``` ## TODO - Remove secrets from config ## Misc ### Applying patches ```bash talosctl patch machineconfig -p @argocd.yaml -n 192.168.0.0 ``` ### Reset node ```bash talosctl reset --system-labels-to-wipe EPHEMERAL,STATE --reboot -n 192.168.0.0 ``` ### ArgoCD default login User: admin, password can be retrieved with (ignore the '%' at the end): ```bash kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ``` ### Ceph default login User: admin on [http://ceph.noxxos.nl](http://ceph.noxxos.nl) ```bash kubectl -n ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo ``` ### Wiping disks for Ceph Start a temporary pod on each node where the disks are: ```bash kubectl run -it --rm \ -n ceph \ --image quay.io/ceph/ceph:v19.2.2 \ --privileged \ --overrides='{"spec": { "nodeSelector": {"kubernetes.io/hostname": "master3"}}}' fix ``` Search for the correct disk with `blkid`, set `DISK=/dev/sdX`, then run (some of) the following commands: ```bash ceph-volume lvm zap /dev/sdX --destroy wipefs -a /dev/sdX # Zap the disk to a fresh, usable state (zap-all is important, b/c MBR has to be clean) sgdisk --zap-all $DISK # Wipe portions of the disk to remove more LVM metadata that may be present dd if=/dev/zero of="$DISK" bs=1K count=200 oflag=direct,dsync seek=0 # Clear at offset 0 dd if=/dev/zero of="$DISK" bs=1K count=200 oflag=direct,dsync seek=$((1 * 1024**2)) # Clear at offset 1GB dd if=/dev/zero of="$DISK" bs=1K count=200 oflag=direct,dsync seek=$((10 * 1024**2)) # Clear at offset 10GB dd if=/dev/zero of="$DISK" bs=1K count=200 oflag=direct,dsync seek=$((100 * 1024**2)) # Clear at offset 100GB dd if=/dev/zero of="$DISK" bs=1K count=200 oflag=direct,dsync seek=$((1000 * 1024**2)) # Clear at offset 1000GB # SSDs may be better cleaned with blkdiscard instead of dd blkdiscard $DISK # Inform the OS of partition table changes partprobe $DISK ``` ### Certificate lifetimes Talos Linux automatically manages and rotates all server side certificates for etcd, Kubernetes, and the Talos API. Note however that the kubelet needs to be restarted at least once a year in order for the certificates to be rotated. Any upgrade/reboot of the node will suffice for this effect. You can check the Kubernetes certificates with the command `talosctl get KubernetesDynamicCerts -o yaml` on the controlplane. Client certificates (talosconfig and kubeconfig) are the user’s responsibility. Each time you download the kubeconfig file from a Talos Linux cluster, the client certificate is regenerated giving you a kubeconfig which is valid for a year. The talosconfig file should be renewed at least once a year, using the `talosctl config new` command.