Skip to content

Cluster API - AWS (Hosted Control Plane)#

This example demonstrates how k0smotron can be used with CAPA (Cluster API Provider Amazon Web Services) to deploy a cluster with hosted control plane and workers in AWS.

Prerequisites#

Before starting this example, ensure that you have met the general prerequisites. In addition to those, you should also have appropriate AWS credentials available and the AWS CLI configured on your local machine.

Prepare the AWS infra provider#

Before launching a cluster, it's crucial to set up your infrastructure provider. Each provider has its unique prerequisites and configuration steps.

Follow the AWS Provider installation guide for detailed steps.

Creating a child cluster#

Once all the controllers are up and running, you can apply the cluster manifests containing the specifications of the cluster you want to provision.

k0smotron in standalone mode is only able to work with externally managed cluster infrastructure.

This is because in CAPA there is no way to disable it to provision all control plane related infrastructure (VPC, ELB, etc.). This also renders k0smotron unable to dynamically edit the AWSCluster API endpoint details. Make sure your VPC and subnets you are planning to use fullfill the needed prerequisites. This limitation is not applicable to the clusters on the VMs.

Here is an example:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: k0s-aws-test
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks: [10.244.0.0/16]
    services:
      cidrBlocks: [10.96.0.0/12]
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: K0smotronControlPlane # This tells that k0smotron should create the controlplane
    name: k0s-aws-test-cp
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSCluster
    name: k0s-aws-test
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: K0smotronControlPlane # This is the config for the controlplane
metadata:
  name: k0s-aws-test-cp
spec:
  version: v1.27.2-k0s.0
  persistence:
    type: emptyDir
  service:
    type: LoadBalancer
    apiPort: 6443
    konnectivityPort: 8132
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSCluster
metadata:
  name: k0s-aws-test
  namespace: default
  annotations:
    cluster.x-k8s.io/managed-by: k0smotron # This marks the base infra to be self managed. The value of the annotation is irrelevant, as long as there is a value.
spec:
  region: eu-central-1
  sshKeyName: ssh-key
  network:
    vpc:
      id: vpc-12345678901234567 # Machines will be created in this VPC
    subnets:
      - id: subnet-099730c9ea2e42134 # Machines will be created in this Subnet
        availabilityZone: eu-central-1a
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: k0s-aws-test-md
  namespace: default
spec:
  clusterName: k0s-aws-test
  replicas: 1
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: k0s-aws-test
      pool: worker-pool-1
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: k0s-aws-test
        pool: worker-pool-1
    spec:
      clusterName: k0s-aws-test
      failureDomain: eu-central-1a
      bootstrap:
        configRef: # This triggers our controller to create cloud-init secret
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: K0sWorkerConfigTemplate
          name: k0s-aws-test-machine-config
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSMachineTemplate
        name: k0s-aws-test-mt
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachineTemplate
metadata:
  name: k0s-aws-test-mt
  namespace: default
spec:
  template:
    spec:
      ami:
        # Replace with your AMI ID
        id: ami-0989fb15ce71ba39e # Ubuntu 22.04 in eu-central-1 
      instanceType: t3.large
      iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io # Instance Profile created by `clusterawsadm bootstrap iam create-cloudformation-stack`
      cloudInit:
        # Makes CAPA use k0s bootstrap cloud-init directly and not via SSM
        # Simplifies the VPC setup as we do not need custom SSM endpoints etc.
        insecureSkipSecretsManager: true
      subnet:
        # Make sure this matches the failureDomain in the Machine, i.e. you pick the subnet ID for the AZ
        id: subnet-099730c9ea2e42134
      additionalSecurityGroups:
        - id: sg-01ce46c31291e3447 # Needs to be belong to the subnet
      sshKeyName: jhennig-key
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: K0sWorkerConfigTemplate
metadata:
  name: k0s-aws-test-machine-config
spec:
  template:
    spec:
      version: v1.27.2+k0s.0
      # More details of the worker configuration can be set here
---

As we are using self-managed infrastructure we need to manually mark the infrastructure ready. This can be accomplished using the following command: kubectl patch AWSCluster k0s-aws-test --type=merge --subresource status --patch 'status: {ready: true}'.

After applying the manifests to the management cluster and confirming the infrastructure readiness, allow a few minutes for all components to provision. Once complete, your command line should display output similar to this:

% kubectl get cluster,machine
NAME                                    PHASE         AGE   VERSION
cluster.cluster.x-k8s.io/k0s-aws-test   Provisioned   16m   

NAME                                         CLUSTER        NODENAME   PROVIDERID                                 PHASE         AGE   VERSION
machine.cluster.x-k8s.io/k0s-aws-test-md-0   k0s-aws-test              aws:///eu-central-1a/i-05f2de7da41dc542a   Provisioned   16m   

You can also check the status of the cluster deployment with clusterctl:

% clusterctl describe cluster  
NAME                                                   READY  SEVERITY  REASON                    SINCE  MESSAGE          
Cluster/k0s-aws-test                                   True                                       25m                      
├─ClusterInfrastructure - AWSCluster/k0s-aws-test                                                                             
├─ControlPlane - K0smotronControlPlane/k0s-aws-test-cp                                                                           
└─Workers                                                                                                                  
  └─Other

Networking Options#

k0smotron, running in a management cluster in AWS, supports flexible networking options, allowing you to choose between Network Load Balancers (NLB) and Classic Elastic Load Balancers (ELB) based on your requirements for exposing the Control Planes.

If you prefer using an NLB instead of ELB, you must specify annotations for the Service in the k0smotronControlPlane. These annotations guide the AWS Cloud Controller Manager (CCM) or the AWS Load Balancer Controller to create the respective services.

 [...] 
  service:
    type: LoadBalancer
    apiPort: 6443
    konnectivityPort: 8132
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

For scenarios involving Classic ELBs or NLBs without special options, the AWS CCM can be utilized.

Internal NLB#

If you aim to use the NLB and set the schema to internal, the target group attribute preserve_client_ip.enabled=false is required due to "hairpinning" or "NAT loopback". In such cases, the AWS CCM cannot be used because it doesn't support setting Target Group Attributes. Therefore, the AWS Load Balancer Controller, which has the ability to set Target Group Attributes, becomes necessary. Follow this guide to install the AWS Load Balancer Controller.

 [...] 
  service:
    type: LoadBalancer
    apiPort: 6443
    konnectivityPort: 8132
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "external" # AWS Loadbalancer Controller creates a NLB when type is "external"
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false

Note: Please make sure that the Security Group does allow the access to the NLB on port 6443 and 8132 from the management cluster nodes. This access is crucial for Cluster API (CAPI), Cluster API Provider for AWS (CAPA), and k0smotron, as they require access to the Control Plane API. Additionally, the port for the Konnectivity service need to be accessible from worker nodes.

Accessing the workload cluster#

To access the child cluster we can get the kubeconfig for it with clusterctl get kubeconfig k0s-aws-test. You can then save it to disk and/or import to your favorite tooling like Lens.

Deleting the cluster#

For cluster deletion, do NOT use kubectl delete -f my-aws-cluster.yaml as that will result in orphan AWS resources. Instead, delete the top level Cluster object. This approach ensures the proper sequence in deleting all child resources, effectively avoiding orphan resources.