Skip to content

MachineHealthChecks with k0smotron#

k0smotron provides built-in support for MachineHealthChecks (MHC), a core Cluster API feature that automatically detects and remediates unhealthy control plane machines.

Overview#

k0smotron's control plane controller automatically handles the remediation process when machines are marked as unhealthy by MHC.

Read more about MachineHealthChecks in the Cluster API documentation.

How it works#

When a MachineHealthCheck detects an unhealthy control plane machine:

  1. The MHC controller marks the machine as unhealthy
  2. k0smotron's control plane controller detects this condition
  3. The controller safely deletes the unhealthy machine
  4. A new machine is automatically created to replace it

Prerequisites#

  • A k0smotron control plane with at least 2 replicas (required for safe remediation)
  • MachineHealthCheck controller running in your management cluster

Example Configuration#

Here's a simple example of how to set up MachineHealthChecks for a k0smotron control plane:

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineHealthCheck
metadata:
  name: k0smotron-controlplane-mhc
  namespace: default
spec:
  clusterName: my-cluster
  selector:
    matchLabels:
      cluster.x-k8s.io/control-plane: "true"
  unhealthyConditions:
  - type: Ready
    status: Unknown
    timeout: 300s
  - type: Ready
    status: "False"
    timeout: 300s
  nodeStartupTimeout: 10m

Safety Features#

k0smotron includes several safety mechanisms to prevent cluster disruption:

  • Minimum replicas: Remediation only occurs when there are more than 1 control plane replica
  • No concurrent operations: Waits for provisioning/deletion operations to complete
  • Graceful deletion: Properly removes machines from the k0s cluster before deletion

Retry limits#

When a Machine created to replace an unhealthy one fails as well, its remediation counts as a retry of the same remediation sequence. spec.remediation on the K0sControlPlane controls those retries.

  • minHealthyPeriodSeconds is how long after a Machine is remediated the remediation of its replacement still counts as a retry. A later remediation starts a new sequence and resets the retry count. It defaults to 3600, and 0 makes every remediation start a new sequence.
  • retryPeriodSeconds is the minimum time between a remediation and the retry that follows it. A retry inside the period is deferred and the Machine reports RemediationDeferred. A retry deferred past minHealthyPeriodSeconds starts a new sequence instead, so keep it below that. No retry period applies when it is not set.
  • maxRetry is how many retries one sequence may make. A retry past it is refused and the Machine reports CannotBeRemediated until minHealthyPeriodSeconds has passed since the previous remediation, when it is remediated as the start of a new sequence. 0 or a negative value refuses every retry, and retries are unlimited when it is not set.
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: K0sControlPlane
metadata:
  name: my-cluster
spec:
  remediation:
    minHealthyPeriodSeconds: 3600
    retryPeriodSeconds: 300
    maxRetry: 3

The most recent remediation and its retry count are reported in status.lastRemediation.

Best Practices#

  • Always use at least 3 control plane replicas in production
  • Set appropriate timeouts based on your infrastructure
  • Monitor remediation events and adjust thresholds as needed
  • Test remediation in non-production environments first