Skip to content

Cluster API - Control Plane Bootstrap provider#

k0smotron can act as a control plane bootstrap provider via usage of K0sControlPlane CRDs.

When creating a Cluster with Cluster API you typically need to create a Cluster object. With k0smotron there needs to be a link to the control plane provider K0sControlPlane:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: cp-test
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - 10.244.0.0/16
    services:
      cidrBlocks:
        - 10.96.0.0/12
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: K0sControlPlane
    name: cp-test

Next we need to provide the configuration for the actual K0sControlPlane and machineTemplate:

---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: K0sControlPlane
metadata:
  name: cp-test
spec:
  replicas: 3
  k0sConfigSpec:
    k0s:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: ClusterConfig
      metadata:
        name: k0s
      spec:
        api:
          extraArgs:
            anonymous-auth: "true" # anonymous-auth=true is needed for k0s to allow unauthorized health-checks on /healthz 
  machineTemplate:
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: DockerMachineTemplate
      name: cp-test-machine-template
      namespace: default
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerMachineTemplate
metadata:
  name: cp-test-machine-template
  namespace: default
spec:
  template:
    spec: {}

By applying this yaml, k0smotron will create 3 machines based on the MachineTemplate configuration, installs k0s with the role controller on each machine and bootstraps the k0s control plane.

For a full reference on K0sControlPlane configurability see the reference docs.

Downscaling the control plane#

WARNING: Downscaling is a dangerous operation and should only be done if you know what you are doing.

Kubernetes using etcd as its backing store. It's crucial to have a quorum of etcd nodes available at all times. Always run etcd as a cluster of odd members.

When downscaling the control plane, you need firstly to deregister the node from the etcd cluster. This can be done by running the following command on the node that will be removed:

k0s etcd leave

NOTE: k0smotron gives node names sequentially and on downscaling it will remove the "latest" nodes. For instance, if you have k0smotron-test cluster of 5 nodes and you downscale to 3 nodes, the nodes k0smotron-test-3 and k0smotron-test-4 will be removed.

After removing members from etcd cluster, you can simply edit the K0sControlPlane object and change the spec.replicas field to the desired number of replicas. k0smotron will then automatically scale down the control plane to the desired number of replicas.

Running workloads on the control plane#

By default, k0s and k0smotron don't run kubelet and any workloads on control plane nodes. But you can enable it by adding --enable-worker flag to the spec.k0sConfigSpec.args in the K0sControlPlane object. This will enable the kubelet on control plane nodes and allow you to run workloads on them.

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: K0sControlPlane
metadata:
  name: docker-test
spec:
  replicas: 1
  k0sConfigSpec:
    args:
      - --enable-worker
      - --no-taints # disable default taints
  machineTemplate:
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: DockerMachineTemplate
      name: docker-test-cp-template
      namespace: default
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerMachineTemplate
metadata:
  name: docker-test-cp-template
  namespace: default
spec:
  template:
    spec: {}

Note: Controller nodes running with --enable-worker are assigned node-role.kubernetes.io/master:NoExecute taint automatically. You can disable default taints using --no-taints parameter.

Client connection tunneling#

k0smotron supports client connection tunneling to the child cluster's control plane nodes. This is useful when you want to access the control plane nodes from a remote location. To enable tunneling, you need to set spec.k0sConfigSpec.tunneling.enabled to true in the K0sControlPlane object.

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: K0sControlPlane
metadata:
  name: docker-test
spec:
  replicas: 1
  k0sConfigSpec:
    tunneling:
      enabled: true
      mode: tunnel # Tunneling mode: tunnel or proxy (default: tunnel)

K0smotron supports two tunneling modes: tunnel and proxy. You can set the tunneling mode using spec.k0sConfigSpec.tunneling.mode field. The default mode is tunnel.

K0smotron will create a kubeconfig file for the tunneling client in the K0sControlPlane object's namespace. You can find the kubeconfig file in the <cluster-name>-<mode>-kubeconfig secret. You can use this kubeconfig file to access the control plane nodes from a remote location.

Note: Parent cluster's worker nodes must be accessible from the child cluster's nodes. You can use spec.k0sConfigSpec.tunneling.serverAddress to set the address of the parent cluster's node or load balancer. If you don't set this field, k0smotron will use the random worker node's address as the default address.

Currently, k0smotron supports only NodePort service type for tunneling. You can set the tunneling service port using spec.k0sConfigSpec.tunneling.tunnelingNodePort field. The default port is 31443.