Deploying my static site with Kubernetes (k3s)

The notes below reflect my mental model and personal difficulties setting up Kubernetes and may contain inaccuracies.

Kubernetes basics for developers who know docker

K8s abstracts literally everything you know and love about docker. Instead of directly creating containers we create Pods that contain 1 or more containers. Instead of exposing ports on a container we create Services that represent how you connect to pods. Services by default don't expose your pods to the outside world, so you need to create an Ingress resource to describe how connections from a host reach a service.

Simply put: network request -> Ingress -> Service -> Pod

Generally you wouldn't provision pods directly, rather you describe Deployments that create replicated pods which spin up automatically so the lifecycle of a request looks more like:

network request -> Ingress -> Service -> Deployment [Round-robin selection] -> Pod

First steps

• Unblock 6443 for remote access with kubectl
• Install k3s on your server with their shell script at https://get.k3s.io
  • Probably need docker and stuff
• Copy /etc/rancher/k3s/k3s.yaml from the server to your localmachine:~/.kube/config
• Replace server IP with your node IP/hostname in the config

Kubernetes Authorization

In Kubernetes, roles and role bindings are used to control access to resources and actions within a namespace, while cluster roles and cluster role bindings are used to control access to resources and actions across the entire cluster.

A brief overview of the differences between these resources:

• Role represents a set of permissions that can be granted to users or service accounts within a single namespace.
• RoleBinding binds a Role to a user or group of users within a namespace, allowing the permissions defined in the Role to be granted to the specified users.
• ClusterRole represents a set of permissions that can be granted to users or service accounts within a cluster. ClusterRoles are global in scope and can be used to grant permissions across all namespaces in the cluster.
• ClusterRoleBinding binds a ClusterRole to a user or group of users within a cluster, allowing the permissions defined in the ClusterRole to be granted to the specified users.

In general, you would use roles and role bindings to grant permissions to users within a specific namespace, and you would use cluster roles and cluster role bindings to grant permissions to users across the entire cluster.

Cert Manager

Really cool tool to provision and renew TLS certs for your hosts but there's a lot of idiosyncrasies that I messed up and spent hours resolving.

If the cluster issuer needs secrets (API tokens for DNS challenges) then the secrets must reside in the same namespace as cert-manager.

Steps to provision TLS with DNS challenge for Cloudflare

1. Create a ClusterIssuer (works across all namespaces)
2. Create a Secret with Cloudflare API token and deploy in cert-manager namespace
3. Annotate ingress with cert-manager.io/cluster-issuer: clusterIssuerName
   • This will automatically generate Certificate resources
   • Alternatively create a Certificate and you don't need to annotate the Ingress

If you wish to use this annotation method to serve a specific host (e.g. foo.example.com) with a certificate at example.com which contains the SAN *.example.com then see the following note from the documentation.

Note: dnsNames take an exact match and do not resolve wildcards, meaning the following Issuer will not solve for DNS names such as foo.example.com. Use the dnsZones selector type to match all subdomains within a zone.

Don't trust yourself and assume that you will get this right on the first try, you're probably going to fuck it up and bust the rate limiting for your certificate authority. Use a staging environment if available to provision your certificates. Even if it succeeds make sure that you're not regenerating certs when you redeploy ingress, restart the cluster, etc.

Traefik Ingress Controller

Exposing the traefik dashboard k -n kube-system port-forward traefik-66c46d954f-fkwdf 8080:9000