Containers

Managing Pod Scheduling Constraints and Groupless Node Upgrades with Karpenter in Amazon EKS

Overview

Karpenter is a high-performance Kubernetes cluster autoscaler that can help you autoscale your groupless nodes by letting you schedule layered constraints using the Provisioner API. Karpenter also makes node upgrades easy through the node expiry TTL value ttlSecondsUntilExpired.

This blog post will walk you through all of the steps to make this possible, and assumes you have an in-depth understanding of Amazon Elastic Kubernetes Service (Amazon EKS) and Karpenter.

Karpenter is an open-source node provisioning project that automatically provisions new nodes in response to unschedulable pods. It observes the aggregate resource requests of unscheduled pods and makes decisions to launch new nodes and terminate them to reduce scheduling latencies as well as the infrastructure costs of sending commands to the underlying cloud provider. Karpenter then launches the nodes with minimal compute resources to fit the unschedulable pods for efficient bin-packing. It works in tandem with the Kubernetes scheduler to bind the unschedulable pods to the new nodes that are provisioned.

Note: Karpenter is designed to be cloud-provider agnostic but currently only supports AWS.

Diagram of pods using Karpenter to optimize capacity

Why Karpenter

Kubernetes users needed to dynamically adjust the compute capacity of their clusters to support applications using Amazon EC2 Auto Scaling groups and the Kubernetes Cluster Autoscaler before the launch of Karpenter. Some of the challenges with Cluster Autoscaler include significant deployment latency because many pods must wait for a node to scale up before they can be scheduled. Nodes can take multiple minutes to become available as Cluster Autoscaler does not bind pods to nodes, and scheduling decisions are made by the kube-scheduler. This results in a longer wait for the nodes to become available, and it can increase pod scheduling latency for critical workloads.

Simplify capacity management with groupless autoscaling

One of the main objectives of Karpenter is to simplify the management of capacity. If you are familiar with other autoscalers, you will notice Karpenter takes a different approach referred to as “groupless autoscaling.” Traditionally, we have used the concept of a node group as the element of control that defines the characteristics of the capacity provided (for example, On-Demand, EC2 Spot Instance, or GPU nodes) and that controls the desired scale of the group in the cluster. In AWS, the implementation of a node group matches with Autoscaling groups. Over time, clusters using this paradigm, which run different types of applications requiring different capacity types, end up with a complex configuration and operational model where node groups must be defined and provided in advance.

Configuring provisioners

Karpenter’s job is to add nodes to handle unschedulable pods (pods with the status condition Unschedulable=True set by the kube-scheduler), schedule pods on those nodes, and remove the nodes when they are not needed. To configure Karpenter, you create provisioners that define how Karpenter manages unschedulable pods and expires nodes.

Provisioner custom resourcedefines the constraints for provisioning the types of nodes and the attributes of the newly provisioned nodes, such as TTL for removing the empty nodes. The provisioner lets you use well-known Kubernetes labels and taints that limit the pods that can run on nodes that Karpenter creates.

It also allows the pods to request nodes based on instance types, architectures, OS, or other attributes by adding specifications to Kubernetes pod deployments. This lets the pod scheduling constraints like Resource requests, Node selection, Node affinity, and Topology spread fall within the provisioner’s constraints for the pods to get deployed on the Karpenter-provisioned nodes. If not, the pods will not deploy.

In many scenarios, a single provisioner can satisfy all the requirements and can use the scheduling constraints with provisioner API and pods. This achieves the use case of different teams having different constraints for running their workloads (such as one team can use only nodes in a specific Availability Zone and other teams can use Arm64 hardware nodes), for billing purposes, having different deprovisioning requirements, and so on.

Use cases for provisioner API constraints

The concept of layered constraints is key for using Karpenter. With no constraints defined in provisioners, and none requested from pods being deployed, Karpenter chooses from the entire universe of features available in the Amazon EC2 fleet. Nodes can be created using any instance type and run in any Zone.

However, for specific requirements such as choosing an instance type or Availability Zones, we can tighten the constraints defined in a provisioner by defining additional scheduling constraints in the pod spec. It also helps when you require certain kinds of processors or other hardware.

Upgrading nodes

A straightforward way to upgrade nodes is to set ttlSecondsUntilExpired. Worker nodes will be terminated after a set period of time and replaced with newer nodes.

Walkthrough

In this section, we’ll provision an Amazon EKS cluster, deploy Karpenter, deploy a sample application, and demonstrate node scaling with Karpenter. We will also look at the process of deploying constraints with pods in line to requirements of provisioner API for different application workloads or multiple teams needing different instance capacity for their application.

Prerequisites

Karpenter deployment tasks

Create an Amazon EKS cluster and node group. Then set up Karpenter and deploy Provisioner API.

  1. Set the following environment variables:
    export CLUSTER_NAME=karpenter-demo
    export AWS_DEFAULT_REGION=ap-south-1
    AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
  2. Create a cluster with eksctl. This example configuration file specifies a basic cluster with one initial node and sets up an IAM OIDC provider for the cluster to enable IAM roles for pods.
    • Note: For an existing EKS cluster, you can determine whether you have one or need to create one in Create an IAM OIDC provider for your cluster.
      eksctl create cluster -f - << EOF
      ---
      apiVersion: eksctl.io/v1alpha5
      kind: ClusterConfig
      metadata:
        name: ${CLUSTER_NAME}
        region: ${AWS_DEFAULT_REGION}
        version: "1.20"
      managedNodeGroups:
        - instanceType: m5.large
          amiFamily: AmazonLinux2
          name: ${CLUSTER_NAME}-ng
          desiredCapacity: 1
          minSize: 1
          maxSize: 2
      iam:
        withOIDC: true
      EOF
      
      export CLUSTER_ENDPOINT="$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.endpoint" --output text)"
  3. Create subnet tags kubernetes.io/cluster/$CLUSTER_NAME.
    • Karpenter discovers subnets tagged kubernetes.io/cluster/$CLUSTER_NAME. Add this tag to associated subnets of your cluster. Retrieve the subnet IDs and tag them with the cluster name.
      SUBNET_IDS=$(aws cloudformation describe-stacks \
       --stack-name eksctl-${CLUSTER_NAME}-cluster \
       --query 'Stacks[].Outputs[?OutputKey==`SubnetsPrivate`].OutputValue' \
       --output text)
       
      aws ec2 create-tags \
       --resources $(echo $SUBNET_IDS | tr ',' '\n') \
       --tags Key="kubernetes.io/cluster/${CLUSTER_NAME}",Value=
      
  4. Create the KarpenterNode IAM role.
    Instances launched by Karpenter must run with an InstanceProfile that grants permissions necessary to run containers and configure networking. Karpenter discovers the InstanceProfile using the name KarpenterNodeRole-${ClusterName}.

    TEMPOUT=$(mktemp)
    
    curl -fsSL https://karpenter.sh/v0.6.3/getting-started/cloudformation.yaml  > $TEMPOUT \
    && aws cloudformation deploy \
      --stack-name Karpenter-${CLUSTER_NAME} \
      --template-file ${TEMPOUT} \
      --capabilities CAPABILITY_NAMED_IAM \
      --parameter-overrides ClusterName=${CLUSTER_NAME}
  5. Grant access to instances using the profile to connect to the cluster. Add the Karpenter node role to your aws-auth configmap.
    eksctl create iamidentitymapping \
     --username system:node:{{EC2PrivateDNSName}} \
     --cluster ${CLUSTER_NAME} \
     --arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME} \
     --group system:bootstrappers \
     --group system:nodes
    
  6. Create the KarpenterController IAM role. Karpenter requires permissions like launching instances. This will create an AWS IAM role and a Kubernetes service account and associate them using IRSA.
    eksctl create iamserviceaccount \
      --cluster "${CLUSTER_NAME}" --name karpenter --namespace karpenter \
      --role-name "${CLUSTER_NAME}-karpenter" \
      --attach-policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}" \
      --role-only \
      --approve
    
    export KARPENTER_IAM_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"
    
  7. Install Karpenter Helm chart.
    helm repo add karpenter https://charts.karpenter.sh
    helm repo update
    
    helm upgrade --install --namespace karpenter --create-namespace \
      karpenter karpenter/karpenter \
      --version v0.6.3 \
      --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
      --set clusterName=${CLUSTER_NAME} \
      --set clusterEndpoint=${CLUSTER_ENDPOINT} \
      --set aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
      --wait # for the defaulting webhook to install before creating a Provisioner
    
  8. Enable debug logging (optional).
    kubectl patch configmap config-logging -n karpenter --patch '{"data":{"loglevel.controller":"debug"}}'
  9. Deploy the provisioner and application pods with layered constraints by applying the following Karpenter provisioner spec. It has the requirements for architecture type (arm64 & amd64), capacity type (Spot & On-demand), and taints for GPU-based use cases.
    cat <<EOF | kubectl apply -f -
    apiVersion: karpenter.sh/v1alpha5
    kind: Provisioner
    metadata:
      name: default
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
        - key: "kubernetes.io/arch" 
          operator: In
          values: ["arm64", "amd64"]
      limits:
        resources:
          cpu: 1000
      provider:    
        subnetSelector:
          kubernetes.io/cluster/$CLUSTER_NAME: '*'
        securityGroupSelector:
          kubernetes.io/cluster/$CLUSTER_NAME: '*'  
      ttlSecondsAfterEmpty: 30
    EOF
    
  10. Run the application deployment on a specific capacity, instance type, hardware, and Availability Zone using pod scheduling constraints.

Sample deployment

In the following sample deployment, we define the nodeSelector with topology.kubernetes.io/zone to choose an Availability Zone and on-demand arm64 instance with karpenter.sh/capacity-type and kubernetes.io/arch: arm64 and specific instance type node.kubernetes.io/instance-type so that new nodes can be launched by Karpenter using the following pod scheduling constraints.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 0
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      nodeSelector:
        node.kubernetes.io/instance-type: r6gd.xlarge
        karpenter.sh/capacity-type: on-demand
        topology.kubernetes.io/zone: ap-south-1a
        kubernetes.io/arch: arm64
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.5
          resources:
            requests:
              cpu: 1
EOF
  1. Scale the above deployment to see the node scaling via Karpenter. It will choose the previous configuration from the Amazon EC2 fleet via the createFleet API for the application pods.kubectl scale deployment inflate --replicas 3
  2. Review the Karpenter pod logs for events and more details.kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
    • Example snippet of the logs:
      eksadmin:~/environment $ kubectl logs -f -n karpenter -l app.kubernetes.io/name*=*karpenter -c controller
      ...
      ...
      2022-02-18T10:48:15.913Z        INFO    controller.provisioning Batched 3 pods in 1.0555487s    {"commit": "fd19ba2", "provisioner": "default"}
      2022-02-18T10:48:15.918Z        INFO    controller.provisioning Computed packing of 1 node(s) for 3 pod(s) with instance type option(s) [r6gd.xlarge]   {"commit": "fd19ba2", "provisioner": "default"}
      2022-02-18T10:48:16.004Z        DEBUG   controller.provisioning Discovered kubernetes version 1.20      {"commit": "fd19ba2", "provisioner": "default"}
      2022-02-18T10:48:16.042Z        DEBUG   controller.provisioning Discovered ami-06a248526462b970e for query /aws/service/eks/optimized-ami/1.20/amazon-linux-2-arm64/recommended/image_id        {"commit": "fd19ba2", "provisioner": "default"}
      2022-02-18T10:48:17.782Z        INFO    controller.provisioning Launched instance: i-034b68cf08d334acf, hostname: ip-192-168-135-148.ap-south-1.compute.internal, type: r6gd.xlarge, zone: ap-south-1a, capacityType: on-demand {"commit": "fd19ba2", "provisioner": "default"}
      2022-02-18T10:48:17.821Z        INFO    controller.provisioning Bound 3 pod(s) to node ip-192-168-135-148.ap-south-1.compute.internal   {"commit": "fd19ba2", "provisioner": "default"}
      2022-02-18T10:48:17.821Z        INFO    controller.provisioning Waiting for unschedulable pods  {"commit": "fd19ba2", "provisioner": "default"}
  3. Validate the application pods with the following command and the same will be in a Running state.
    kubectl get node -L node.kubernetes.io/instance-type,kubernetes.io/arch,karpenter.sh/capacity-type
    
    kubectl get pods -o wide

    Example snippet of the node output and pods output.

    eksadmin:~/environment $ kubectl get node -L node.kubernetes.io/instance-type,kubernetes.io/arch,karpenter.sh/capacity-type
    NAME                                             STATUS   ROLES    AGE   VERSION               INSTANCE-TYPE   ARCH    CAPACITY-TYPE
    ip-192-168-135-148.ap-south-1.compute.internal   Ready    <none>   79s   v1.20.11-eks-f17b81   r6gd.xlarge     arm64   on-demand
    ip-192-168-4-178.ap-south-1.compute.internal     Ready    <none>   22m   v1.20.11-eks-f17b81   m5.large        amd64   
    eksadmin:~/environment $ 
    eksadmin:~/environment $ kubectl get pods -o wide
    NAME                       READY   STATUS    RESTARTS   AGE   IP               NODE                                             NOMINATED NODE   READINESS GATES
    inflate-56c45f4967-6x9h9   1/1     Running   0          87s   192.168.159.41   ip-192-168-135-148.ap-south-1.compute.internal   <none>           <none>
    inflate-56c45f4967-cffdd   1/1     Running   0          87s   192.168.145.73   ip-192-168-135-148.ap-south-1.compute.internal   <none>           <none>
    inflate-56c45f4967-xq4tw   1/1     Running   0          87s   192.168.130.79   ip-192-168-135-148.ap-south-1.compute.internal   <none>           <none>
    eksadmin:~/environment $

    We can see Karpenter applied layered constraints to launch nodes that satisfy multiple scheduling constraints of a workload, like instance type, specific Availability Zone, and hardware architecture via Karpenter.

Groupless Node upgrades

When using the node groups (self-managed or managed) with an EKS cluster, and as part of upgrading the worker nodes to a newer version of Kubernetes, we would have to rely on either migrating to a new node group for self-managed or launching a new autoscaling group of worker nodes for a managed node group, as mentioned in managed node group update behavior. Whereas with the Karpenter groupless autoscaling the upgrade of nodes works with the expiry time-to-live value.

Karpenter Provisioner API has Node Expiry that will allow a node to expire on reaching the expiry time-to-live value (ttlSecondsUntilExpired). The same value is used to upgrade nodes ttlSecondsUntilExpired. The nodes will be terminated after a set period of time, after which they are replaced with newer nodes.

Note: Karpenter supports using custom launch templates. When using a custom launch template, you are taking responsibility for maintaining the launch template, including updating which AMI is used (that is, for security updates). In the default configuration, Karpenter will use the latest version of the EKS optimized AMI, which is maintained by AWS.

  1. Validate the current EKS cluster Kubernetes version with the following command.aws eks describe-cluster --name ${CLUSTER_NAME} | grep -i versionThe following is an example snippet of the previous command:
    eksadmin:~/environment $ aws eks describe-cluster --name karpenter-demo | grep -i version
    "platformVersion": "eks.3",
    "version": "1.20",
    eksadmin:~/environment $
    
  2. Deploy PodDisruptionBudget for your application deployment. PodDisruptionBudget (PDB) limits the number of pods of a replicated application that are down simultaneously from voluntary disruptions.
    cat <<EOF | kubectl apply -f -
    apiVersion: policy/v1beta1
    kind: PodDisruptionBudget
    metadata:
      name: inflate-pdb
    spec:
      minAvailable: 2
      selector:
        matchLabels:
          app: inflate
    EOF
    

    The following is an example snippet of previous PDB and sample application deployment that was configured in an earlier section:

    eksadmin:~/environment $ kubectl get pdb
    NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
    inflate-pdb 2 N/A 1 2m7s
    eksadmin:~/environment $
    eksadmin:~/environment $ kubectl get deploy inflate
    NAME READY UP-TO-DATE AVAILABLE AGE
    inflate 3/3 3 3 39m
    eksadmin:~/environment $
  3. Upgrade the EKS cluster to a newer Kubernetes version via the console or eksctl as mentioned in the EKS documentation. We can see that the cluster was upgraded successfully to 1.21.
    eksadmin:~/environment $ aws eks describe-cluster --name ${CLUSTER_NAME}  | grep -i version                                                                     
                "alpha.eksctl.io/eksctl-version": "0.79.0", 
            "platformVersion": "eks.4", 
            "version": "1.21", 
    eksadmin:~/environment $
    
  4. Checking our workload and node created by Karpenter earlier, we can see that nodes are of version 1.20 as Karpenter used the latest version of the EKS optimized AMI based on the earlier EKS cluster version 1.20.
    eksadmin:~/environment $ kubectl get node -L node.kubernetes.io/instance-type,kubernetes.io/arch,karpenter.sh/capacity-type
    NAME                                             STATUS   ROLES    AGE    VERSION               INSTANCE-TYPE   ARCH    CAPACITY-TYPE
    ip-192-168-135-148.ap-south-1.compute.internal   Ready    <none>   115m   v1.20.11-eks-f17b81   r6gd.xlarge     arm64   on-demand
    ip-192-168-4-178.ap-south-1.compute.internal     Ready    <none>   137m   v1.20.11-eks-f17b81   m5.large        amd64   
    eksadmin:~/environment $ 
    eksadmin:~/environment $ kubectl get pods -o wide                                                                                                         
    NAME                       READY   STATUS    RESTARTS   AGE    IP               NODE                                             NOMINATED NODE   READINESS GATES
    inflate-56c45f4967-6x9h9   1/1     Running   0          116m   192.168.159.41   ip-192-168-135-148.ap-south-1.compute.internal   <none>           <none>
    inflate-56c45f4967-cffdd   1/1     Running   0          116m   192.168.145.73   ip-192-168-135-148.ap-south-1.compute.internal   <none>           <none>
    inflate-56c45f4967-xq4tw   1/1     Running   0          116m   192.168.130.79   ip-192-168-135-148.ap-south-1.compute.internal   <none>           <none>
    eksadmin:~/environment $   
    
  5. Now, let’s reconfigure the provisioner API of Karpenter and append ttlSecondsUntilExpired. This will add the node expiry, which allows the nodes to get terminated and replaced with a new one matching the EKS cluster Kubernetes version 1.21 now.
    cat <<EOF | kubectl apply -f -
    apiVersion: karpenter.sh/v1alpha5
    kind: Provisioner
    metadata:
      name: default
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
        - key: "kubernetes.io/arch" 
          operator: In
          values: ["arm64", "amd64"]
      limits:
        resources:
          cpu: 1000
      provider:    
        subnetSelector:
          kubernetes.io/cluster/$CLUSTER_NAME: '*'
        securityGroupSelector:
          kubernetes.io/cluster/$CLUSTER_NAME: '*'  
      ttlSecondsAfterEmpty: 30
      ttlSecondsUntilExpired: 1800
    EOF
    

    Note: If ttlSecondsUntilExpired is nil, that means that the feature is disabled and nodes will never expire. For an example value, we can configure the node expiry to a value of 30 days as ttlSecondsUntilExpired: 2592000 (# 30 Days = 60 * 60 * 24 * 30 Seconds).

  6. Review the Karpenter pod logs for events and more details.
    kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller

    The following is an example snippet of the logs:

    2022-02-18T12:48:31.289Z        DEBUG   controller.provisioning Discovered kubernetes version 1.21 {"commit": "fd19ba2", "provisioner": "default"}
    2022-02-18T12:48:31.289Z        DEBUG   controller.provisioning Discovered ami-0aa1d81ef621af702 for query /aws/service/eks/optimized-ami/1.21/amazon-linux-2-arm64/recommended/image_id        {"commit": "fd19ba2", "provisioner": "default"}
    2022-02-18T12:48:31.414Z        DEBUG   controller.provisioning Created launch template, Karpenter-karpenter-demo-7876638028496176806   {"commit": "fd19ba2", "provisioner": "default"}
    2022-02-18T12:48:32.732Z        DEBUG   controller.eviction     Did not evict pod default/inflate-56c45f4967-6x9h9 due to PDB violation.     {"commit": "fd19ba2"}
    2022-02-18T12:48:32.740Z        DEBUG   controller.eviction     Did not evict pod default/inflate-56c45f4967-xq4tw due to PDB violation.     {"commit": "fd19ba2"}
    2022-02-18T12:48:33.089Z        INFO    controller.provisioning Launched instance: i-0a3b7cb681485f416, hostname: ip-192-168-132-35.ap-south-1.compute.internal, type: r6gd.xlarge, zone: ap-south-1a, capacityType: on-demand  {"commit": "fd19ba2", "provisioner": "default"}
    2022-02-18T12:48:33.110Z        INFO    controller.provisioning Bound 1 pod(s) to node ip-192-168-132-35.ap-south-1.compute.internal    {"commit": "fd19ba2", "provisioner": "default"}
    2022-02-18T12:48:33.110Z        INFO    controller.provisioning Waiting for unschedulable pods  {"commit": "fd19ba2", "provisioner": "default"}
    2022-02-18T12:48:35.944Z        DEBUG   controller.eviction     Did not evict pod default/inflate-56c45f4967-6x9h9 due to PDB violation.     {"commit": "fd19ba2"}
    2022-02-18T12:48:35.952Z        DEBUG   controller.eviction     Did not evict pod default/inflate-56c45f4967-xq4tw due to PDB violation.     {"commit": "fd19ba2"}
    2022-02-18T12:48:42.354Z        DEBUG   controller.eviction     Did not evict pod default/inflate-56c45f4967-6x9h9 due to PDB violation.     {"commit": "fd19ba2"}
    2022-02-18T12:49:42.451Z        DEBUG   controller.eviction     Evicted pod default/inflate-56c45f4967-6x9h9    {"commit": "fd19ba2"}
    2022-02-18T12:49:42.460Z        INFO    controller.provisioning Bound 1 pod(s) to node ip-192-168-132-35.ap-south-1.compute.internal    {"commit": "fd19ba2", "provisioner": "default"}
    2022-02-18T12:49:42.467Z        DEBUG   controller.eviction     Did not to evict pod default/inflate-56c45f4967-xq4tw due to PDB violation.     {"commit": "fd19ba2"}
    2022-02-18T12:49:52.510Z        DEBUG   controller.eviction     Evicted pod default/inflate-56c45f4967-xq4tw    {"commit": "fd19ba2"}
    2022-02-18T12:49:57.747Z        INFO    controller.termination  Deleted node    {"commit": "fd19ba2", "node": "ip-192-168-135-148.ap-south-1.compute.internal"}
    2022-02-18T12:49:57.765Z        INFO    controller.provisioning Bound 1 pod(s) to node ip-192-168-132-35.ap-south-1.compute.internal    {"commit": "fd19ba2", "provisioner": "default"}
    2022-02-18T12:49:57.888Z        INFO    controller.provisioning Waiting for unschedulable pods  {"commit": "fd19ba2", "provisioner": "default"}
    

    Note: In the previous logs, we can see that PodDisruptionBudget was respected by Karpenter, and then it Discovered kubernetes version 1.21, used the latest version of the EKS optimized AMI for 1.21, and launched a new node for the workload. Later, the old node was cordoned, drained, and deleted by Karpenter.

    If we validate the application pods with the following commands, we can see that Karpenter launched nodes are upgraded to 1.21, same as that of the EKS cluster Kubernetes version.

    kubectl get node -L node.kubernetes.io/instance-type,kubernetes.io/arch,karpenter.sh/capacity-type
    
    kubectl get pods -o wide
    

    The following is an example snippet of the node output and pods output:

    eksadmin:~/environment $ kubectl get node -L node.kubernetes.io/instance-type,kubernetes.io/arch,karpenter.sh/capacity-type                                  
    NAME                                            STATUS   ROLES    AGE    VERSION               INSTANCE-TYPE   ARCH    CAPACITY-TYPE
    ip-192-168-132-35.ap-south-1.compute.internal   Ready    <none>   113s   v1.21.5-eks-9017834   r6gd.xlarge     arm64   on-demand
    ip-192-168-4-178.ap-south-1.compute.internal    Ready    <none>   143m   v1.20.11-eks-f17b81   m5.large        amd64   
    eksadmin:~/environment $ 
    eksadmin:~/environment $ kubectl get pods -o wide                                                                                                           
    NAME                       READY   STATUS    RESTARTS   AGE    IP                NODE                                            NOMINATED NODE   READINESS GATES
    inflate-56c45f4967-4cdx6   1/1     Running   0          49s    192.168.129.193   ip-192-168-132-35.ap-south-1.compute.internal   <none>           <none>
    inflate-56c45f4967-gzgbh   1/1     Running   0          39s    192.168.158.104   ip-192-168-132-35.ap-south-1.compute.internal   <none>           <none>
    inflate-56c45f4967-hfqwb   1/1     Running   0          2m2s   192.168.152.242   ip-192-168-132-35.ap-south-1.compute.internal   <none>           <none>
    eksadmin:~/environment $ 
    

    In the previous demonstration, we see that Karpenter respected the PDB and its ability to apply node expiry for upgrading of nodes launched by Karpenter.

    Node expiry can be used as a means of upgrading or repacking nodes so that nodes are retired and replaced with updated versions. See How Karpenter nodes are deprovisioned in the Karpenter documentation for information on using ttlSecondsUntilExpired and ttlSecondsAfterEmpty.

Cleanup

Delete all the provisioners (CRDs) that were created.

kubectl delete provisioner default

Remove Karpenter and delete the infrastructure from your AWS account.

helm uninstall karpenter --namespace karpenter
eksctl delete iamserviceaccount --cluster ${CLUSTER_NAME} --name karpenter --namespace karpenter
aws cloudformation delete-stack --stack-name Karpenter-${CLUSTER_NAME}
aws ec2 describe-launch-templates \
    | jq -r ".LaunchTemplates[].LaunchTemplateName" \
    | grep -i Karpenter-${CLUSTER_NAME} \
    | xargs -I{} aws ec2 delete-launch-template --launch-template-name {}
eksctl delete cluster --name ${CLUSTER_NAME}

Conclusion

Karpenter provides the option to scale nodes quickly and with very little latency. In this blog, we demonstrated how the nodes can be scaled with different options for each use case using Provisioner API by leveraging the well-known Kubernetes labels and taints and using the pod scheduling constraints within the deployment so that Pods get deployed on the Karpenter provisioned nodes. This demonstrates that we can run different types of workloads on different capacities or requirements for each of its use cases. Further, we see the upgrade node behavior for the nodes launched by Karpenter by enabling the node expiry time ttlSecondsUntilExpired with the provisioner API.

Gowtham S

Gowtham S

Gowtham S is a Container Specialist Technical Account Manager at AWS, based out of Bengaluru. Gowtham works with AWS Enterprise Support customers helping them to optimize Kubernetes workloads through pro-active operations reviews. He is passionate about Kubernetes and open-source technologies.

Abhishek Nanda

Abhishek Nanda

Abhishek is a Containers Specialist Solutions Architect at AWS based out of Bengaluru, Karnataka, India with over 7 years of IT experience. He is passionate designing and architecting secure, resilient and cost effective containerized infrastructure and applications.