Veda

The new setup of my homelab will be based on Kubernetes, which will prevent all of my services going down when I need to do physical maintenance of a host.

Services

Core

• Ceph for all storage: cephfs, object storage and block storage
• Nextcloud: file storage interface for the entire family
• Jellyfin: Web based media streaming
• Authentik: Central identification and authentication server
• Nginx reverse proxy
• ACME client: SSL certificate handling
• ArgoCD: Revision control for all Kubernetes configuration
• Homeassistant + Zigbee2mqtt
• Prometheus
• Grafana
• Grafana Loki + FluentD
• Cilium
• Harbor: Container image storage

Nice-to-have

• Jellyseerr: Nice interface to request movies and series
• Sonarr: Automated downloading and handling of series
• Radarr: Automated downloading and handling of movies
• Flaresolverr: Fetching data hidden behind captcha’s
• Torrent client (qBittorrent): To download all the linux ISO’s
• ExternalDNS
• Paperless-ngx

Look-into-later

• Mastodon: federated social platform
• Forgejo: Git platform. Maybe this should not be hosted on the cluster as it will depend on it.
• CloudNativePG: K8s operator for PostgreSQL

Installing

Configuration

export CLUSTER_NAME="veda"
export API_ENDPOINT="https://192.168.0.1:6443"

talosctl gen secrets --output-file secrets.yaml

talosctl gen config             \
    --with-secrets secrets.yaml \
    --output-types talosconfig  \
    --output talosconfig        \
    $CLUSTER_NAME               \
    $API_ENDPOINT

talosctl config merge ./talosconfig

For controlplane nodes:

talosctl gen config \
        --output rendered/master1.yaml                            \
        --output-types controlplane                               \
        --with-secrets secrets.yaml                               \
        --config-patch @nodes/master1.yaml                        \
        --config-patch @patches/argocd.yaml                       \
        --config-patch @patches/cilium.yaml                       \
        --config-patch @patches/scheduling.yaml                   \
        --config-patch @patches/discovery.yaml                    \
        --config-patch @patches/diskselector.yaml                 \
        --config-patch @patches/vip.yaml                          \
        $CLUSTER_NAME                                             \
        $API_ENDPOINT

For worker nodes:

talosctl gen config \
        --output rendered/worker1.yaml                            \
        --output-types controlplane                               \
        --with-secrets secrets.yaml                               \
        --config-patch @nodes/worker1.yaml                        \
        --config-patch @patches/argocd.yaml                       \
        --config-patch @patches/cilium.yaml                       \
        --config-patch @patches/scheduling.yaml                   \
        --config-patch @patches/discovery.yaml                    \
        --config-patch @patches/diskselector.yaml                 \
        $CLUSTER_NAME                                             \
        $API_ENDPOINT

Then change the endpoint in the Talos client configuration:

# ~/.talos/config
context: veda
contexts:
    veda:
        endpoints: 
            - 192.168.0.1
# (...)

Bootstrapping

Apply the configuration to each node:

talosctl apply-config --insecure --file rendered/master1.yaml  --nodes 192.168.0.10

Optionally, check the status.

talosctl -n 192.168.0.10 dashboard

To start the cluster, we need to bootstrap the etcd cluster. This only has to be done for a single node.

talosctl -n 192.168.0.10

Finally, retrieve the kubeconfig, it will merge with ~/.kube/config, if it exists.

talosctl -n 192.168.0.10 kubeconfig

Check nodes:

talosctl get nodes

TODO

• Remove secrets from config

Misc

Applying patches

talosctl patch machineconfig -p @argocd.yaml -n 192.168.0.0

Reset node

talosctl reset --system-labels-to-wipe EPHEMERAL,STATE --reboot -n 192.168.0.0

ArgoCD default login

User: admin, password can be retrieved with (ignore the '%' at the end):

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Certificate lifetimes

Talos Linux automatically manages and rotates all server side certificates for etcd, Kubernetes, and the Talos API. Note however that the kubelet needs to be restarted at least once a year in order for the certificates to be rotated. Any upgrade/reboot of the node will suffice for this effect.

You can check the Kubernetes certificates with the command talosctl get KubernetesDynamicCerts -o yaml on the controlplane.

Client certificates (talosconfig and kubeconfig) are the user’s responsibility. Each time you download the kubeconfig file from a Talos Linux cluster, the client certificate is regenerated giving you a kubeconfig which is valid for a year.

The talosconfig file should be renewed at least once a year, using the talosctl config new command.