meta-virtualization/recipes-containers/kubernetes

k8s: Kubernetes

meta-virtualization provides the recipes and packages suitable for a k8s cluster instance.

For a kubernetes controller:

• packagegroup-k8s-host

For a kubernetes worker/node:

• packagegroup-k8s-node

If kernel issues or missing features are detected, consider adding the "kernel-modules" package to your image (Since the configuration and RDEPENDS may not be correct for your kernel + k8s version).

CNI

The CNI base packages provide core support and are installed by default as dependencies of the kubernetes packages. Minimal configuration and startup are provided, but you will need to apply the CNI configuration of your choice after boot (see below fo an example)

Configure and initialize the host

A convenience script "k8s-init" is provided to do basic setup on the controller node. After the contoller boots, run it for kubeadm setup and other basic configuration.

Once the node is ready ('kubectl get nodes' to check), follow the instructions for copying the token to your home directory, and apply the networking configuration of choice (flannel in the example):

  % mkdir -p $HOME/.kube
  % cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  % chown $(id -u):$(id -g) $HOME/.kube/config

  % kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Configuration and initialize the node/worker

Once the kubernetes node has booted, it is ready to join the cluster. Some basica configuration is done via the packages and a systctl.d configuration snippet.

Join the cluster (substitute your controller ip and token information):

kubeadm join <controller ip>:6443 --token cq8ngi.6m6mgqi9zf08ypc4 --discovery-token-ca-cert-hash sha256:6064ae531c8dad824f9eadff030f83ec84d00796fac75f1adbd343255eb34fd2

Notes:

Memory:

if running under qemu, the default of 256M of memory is not enough, k3s will OOM and exit.

Boot with qemuparams="-m 2048" to boot with 2G of memory (or choose the appropriate amount for your configuration)

CPUs:

Kubernetes needs at least two cpus, so ensure your qemuboot is smp of at least 2, and/or that your hardware has the required capabilties.

Disk:

if using qemu and core-image* you'll need to add extra space in your disks to ensure containers can start. The following in your image recipe, or local.conf would add 2G of extra space to the rootfs:

IMAGE_ROOTFS_EXTRA_SPACE = "2097152"

Example qemux86-64 boot line:

runqemu qemux86-64 nographic kvm slirp qemuparams="-m 2048"

k8s logs can be seen via:

% journalctl -u kubelet

or

% journalctl -xe

Example output from qemux86-64:

If you've lost the join token, you can create a new one, or list existing ones:

root@qemux86-64-7b:~# kubeadm token create --print-join-command
kubeadm join 10.10.10.117:6443 --token dr71zq.y5vi3s2n2antvcej --discovery-token-ca-cert-hash sha256:6064ae531c8dad824f9eadff030f83ec84d00796fac75f1adbd343255eb34fd2

root@qemux86-64-7b:~# kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
cq8ngi.6m6mgqi9zf08ypc4   23h         2021-12-16T16:58:02Z   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token
dr71zq.y5vi3s2n2antvcej   23h         2021-12-16T17:46:28Z   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

root@qemux86-64:~# kubectl get nodes
NAME            STATUS   ROLES                  AGE   VERSION
qemux86-64-7b   Ready    control-plane,master   51m   v1.23.1-rc.0.1+dd1b0a12471310-dirty
qemux86-64-9d   Ready    <none>                 49m   v1.23.1-rc.0.1+dd1b0a12471310-dirty