Skip to content

Cluster API - VMware#

This example demonstrates how k0smotron can be used with CAPV (Cluster API Provider vSphere).

Table of Contents

Setting the scene#

To use Cluster API we will bring three environments: 1. Local machine -- the machine from where some of the commands will be executed 2. Management cluster -- Kubernetes cluster that will be used to control and manage child clusters 3. Child cluster -- k0s cluster that will be spinned up with the help of k0smotron


Before starting this example, ensure that you have met the general prerequisites.

Configure clusterctl on the local machine#

  1. Create the cluster-api configuration file in your development machine.
    mkdir ~/.cluster-api
    vim ~/.cluster-api/clusterctl.yaml
  2. Put values according to your environment in this file:
## -- Controller settings -- ##
VSPHERE_USERNAME: "vi-admin@vsphere.local"                    # The username used to access the remote vSphere endpoint
VSPHERE_PASSWORD: "admin!23"                                  # The password used to access the remote vSphere endpoint

## -- Required workload cluster default settings -- ##
VSPHERE_SERVER: ""                                    # The vCenter server IP or FQDN
VSPHERE_DATACENTER: "SDDC-Datacenter"                         # The vSphere datacenter to deploy the management cluster on
VSPHERE_DATASTORE: "DefaultDatastore"                         # The vSphere datastore to deploy the management cluster on
VSPHERE_NETWORK: "VM Network"                                 # The VM network to deploy the management cluster on
VSPHERE_RESOURCE_POOL: "*/Resources"                          # The vSphere resource pool for your VMs
VSPHERE_FOLDER: "vm"                                          # The VM folder for your VMs. Set to "" to use the root vSphere folder
VSPHERE_TEMPLATE: "ubuntu-1804-kube-v1.17.3"                  # The VM template to use for your management cluster.
CONTROL_PLANE_ENDPOINT_IP: ""                    # the IP that kube-vip is going to use as a control plane endpoint
VIP_NETWORK_INTERFACE: "ens192"                               # The interface that kube-vip should apply the IP to. Omit to tell kube-vip to autodetect the interface.
VSPHERE_TLS_THUMBPRINT: "..."                                 # sha1 thumbprint of the vcenter certificate: openssl x509 -sha1 -fingerprint -in ca.crt -noout
EXP_CLUSTER_RESOURCE_SET: "true"                              # This enables the ClusterResourceSet feature that we are using to deploy CSI
VSPHERE_SSH_AUTHORIZED_KEY: "ssh-rsa AAAAB3N..."              # The public ssh authorized key on all machines in this cluster.
                                                              #  Set to "" if you don't want to enable SSH, or are using another solution.
VSPHERE_STORAGE_POLICY: ""                                    # This is the vSphere storage policy. Set it to "" if you don't want to use a storage policy.
"CPI_IMAGE_K8S_VERSION": "v1.29.0"                            # The version of the vSphere CPI image to be used by the CPI workloads
                                                              #  Keep this close to the minimum Kubernetes version of the cluster being created.
K0S_VERSION: "v1.29.1+k0s.1"
K0S_CP_VERSION: "v1.29.1-k0s.1"
NODE_IPAM_POOL_NAME: "ipam-ip-pool"
  - name: incluster
    type: IPAMProvider

Deploy Cluster API in the management cluster#

From your local machine run the following command to initialize the management cluster with vSphere infrastructure provider and additional IPAM provider:

clusterctl init --infrastructure vsphere --ipam incluster

NOTE: In order to initialize Cluster API on you Kubernetes management cluster, you need to have kubeconfig (or set context with kubectl) to the cluster.

NOTE: IPAM provider is optional here. If you're using network that has DHCP enabled, you can remove IPAM provider initialization.

For more details on Cluster API Provider vSphere see it's docs.

(Optional) IPAM IP pool creation#

  1. Define IP pool that will be used to assign IP addresses to a child cluster machines in capi-ipam.yaml:
kind: InClusterIPPool
  name: ipam-ip-pool
  1. Deploy the IPAM configuration:
    kubectl apply -f capi-ipam.yaml

(Optional) MetalLB as Load Balancer solution in the management cluster#

You may require to deploy MetalLB if you don't have Load Balancer solution in your management cluster. NOTE: Load Balancer service type will be used only for the scenario, when you put you k0s control plane in pods (see Control plane in Pods section)

Classic vSphere does not have a built-in solution that can be leveraged on Kubernetes level to create LoadBalancer service type. In order to make it possible, one of the common solution is to use MetalLB (please find it's docs here)

  1. Configure MetalLB IP address pool which will be used to assing IP addresses from (please adapt it for your network):
kind: IPAddressPool
  name: example-pool
  namespace: metallb-system
  1. Create MetalLB L2advertisement object:
kind: L2Advertisement
  name: example-l2-adv
  namespace: metallb-system
  - example-pool

Operating child clusters#

Once all the controllers are up and running in the management cluster, you can apply the cluster manifests containing the specifications of the child cluster you want to provision.

Generate a child cluster definition using the template#

There are two options for child cluster control plane with k0smotron:

Control plane in Pods:#

If you want to have child cluster control plane in a Pods, use this template:

clusterctl generate cluster my-cluster --control-plane-machine-count 1 --worker-machine-count 1 --from k0scluster-tmpl-cp-in-pods.yaml > my-cluster.yaml

Control plane in VMs:#

If you want to have child cluster control plane in separate VMs, use that template (CP will consist of 3 controllers):

clusterctl generate cluster my-cluster --worker-machine-count 1 --from k0scluster-tmpl-cp-in-vms.yaml > my-cluster.yaml 

Deploy the child clusters#

  1. Deploy the child cluster: kubectl apply -f my-cluster.yaml. You can check the deployment status in the management cluster.

  2. To obtain kubeconfig of the child cluster execute this command: kubectl get secret my-cluster-kubeconfig -o jsonpath='{.data.value}' | base64 -d > ~/.kube/child.conf

Observe the child cluster objects#

# kubectl get cluster,machine,kmc                                                                                                                   

NAME                                  CLUSTERCLASS   PHASE         AGE   VERSION                  Provisioned   13h   

NAME                                                   CLUSTER      NODENAME                      PROVIDERID                                       PHASE         AGE   VERSION                  my-cluster                                 vsphere://4215e794-c281-5cde-8193-95df9521cf08   Provisioned   13h   v1.29.1                  my-cluster                                 vsphere://4215efc3-303e-938d-3787-c4cc6143f722   Provisioned   13h   v1.29.1                  my-cluster                                 vsphere://4215b207-1052-e130-54fe-fd124594ed2b   Provisioned   13h   v1.29.1   my-cluster   my-cluster-md-0-vr6s6-ndl64   vsphere://4215eae5-3456-9bc5-4b56-b00dda520b7d   Running       13h   v1.29.1+k0s.1

You can also check the status of the cluster deployment with clusterctl:

# clusterctl describe cluster my-cluster
NAME                                                        READY  SEVERITY  REASON  SINCE  MESSAGE                             
Cluster/my-cluster                                          True                     13h                                         
├─ClusterInfrastructure - VSphereCluster/my-cluster         True                     13h                                         
├─ControlPlane - K0sControlPlane/my-cluster                                                                                       └─3 Machines...                                           True                     13h    See my-cluster-0, my-cluster-1, ...  
  └─MachineDeployment/my-cluster-md-0                       True                     12h                                         
    └─Machine/my-cluster-md-0-vr6s6-ndl64                   True                     12h                                         
      └─BootstrapConfig - K0sWorkerConfig/my-cluster-htrzb                                                          

Deleting the child cluster#

For cluster deletion, do NOT use kubectl delete -f my-cluster.yaml as that can result in orphan resources. Instead, delete the top level Cluster object. This approach ensures the proper sequence in deleting all child resources, effectively avoid orphan resources.

To do that, you can use the command kubectl delete cluster my-cluster