Overview
Karpenter is a high-performance Kubernetes cluster autoscaler that can help you autoscale your groupless nodes by letting you schedule layered constraints using the Provisioner API. Karpenter also makes node upgrades easy through the node expiry TTL value ttlSecondsUntilExpired.
This blog post will walk you through all of the steps to make this possible, and assumes you have an in-depth understanding of Amazon Elastic Kubernetes Service (Amazon EKS) and Karpenter.
Karpenter is an open-source node provisioning project that automatically provisions new nodes in response to unschedulable pods. It observes the aggregate resource requests of unscheduled pods and makes decisions to launch new nodes and terminate them to reduce scheduling latencies as well as the infrastructure costs of sending commands to the underlying cloud provider. Karpenter then launches the nodes with minimal compute resources to fit the unschedulable pods for efficient bin-packing. It works in tandem with the Kubernetes scheduler to bind the unschedulable pods to the new nodes that are provisioned.
Note: Karpenter is designed to be cloud-provider agnostic but currently only supports AWS.
Why Karpenter
Kubernetes users needed to dynamically adjust the compute capacity of their clusters to support applications using Amazon EC2 Auto Scaling groups and the Kubernetes Cluster Autoscaler before the launch of Karpenter. Some of the challenges with Cluster Autoscaler include significant deployment latency because many pods must wait for a node to scale up before they can be scheduled. Nodes can take multiple minutes to become available as Cluster Autoscaler does not bind pods to nodes, and scheduling decisions are made by the kube-scheduler. This results in a longer wait for the nodes to become available, and it can increase pod scheduling latency for critical workloads.
Simplify capacity management with groupless autoscaling
One of the main objectives of Karpenter is to simplify the management of capacity. If you are familiar with other autoscalers, you will notice Karpenter takes a different approach referred to as “groupless autoscaling.” Traditionally, we have used the concept of a node group as the element of control that defines the characteristics of the capacity provided (for example, On-Demand, EC2 Spot Instance, or GPU nodes) and that controls the desired scale of the group in the cluster. In AWS, the implementation of a node group matches with Autoscaling groups. Over time, clusters using this paradigm, which run different types of applications requiring different capacity types, end up with a complex configuration and operational model where node groups must be defined and provided in advance.
Configuring provisioners
Karpenter’s job is to add nodes to handle unschedulable pods (pods with the status condition Unschedulable=True set by the kube-scheduler), schedule pods on those nodes, and remove the nodes when they are not needed. To configure Karpenter, you create provisioners that define how Karpenter manages unschedulable pods and expires nodes.
Provisioner custom resourcedefines the constraints for provisioning the types of nodes and the attributes of the newly provisioned nodes, such as TTL for removing the empty nodes. The provisioner lets you use well-known Kubernetes labels and taints that limit the pods that can run on nodes that Karpenter creates.
It also allows the pods to request nodes based on instance types, architectures, OS, or other attributes by adding specifications to Kubernetes pod deployments. This lets the pod scheduling constraints like Resource requests, Node selection, Node affinity, and Topology spread fall within the provisioner’s constraints for the pods to get deployed on the Karpenter-provisioned nodes. If not, the pods will not deploy.
In many scenarios, a single provisioner can satisfy all the requirements and can use the scheduling constraints with provisioner API and pods. This achieves the use case of different teams having different constraints for running their workloads (such as one team can use only nodes in a specific Availability Zone and other teams can use Arm64 hardware nodes), for billing purposes, having different deprovisioning requirements, and so on.
Use cases for provisioner API constraints
The concept of layered constraints is key for using Karpenter. With no constraints defined in provisioners, and none requested from pods being deployed, Karpenter chooses from the entire universe of features available in the Amazon EC2 fleet. Nodes can be created using any instance type and run in any Zone.
However, for specific requirements such as choosing an instance type or Availability Zones, we can tighten the constraints defined in a provisioner by defining additional scheduling constraints in the pod spec. It also helps when you require certain kinds of processors or other hardware.
Upgrading nodes
A straightforward way to upgrade nodes is to set ttlSecondsUntilExpired. Worker nodes will be terminated after a set period of time and replaced with newer nodes.
Walkthrough
In this section, we’ll provision an Amazon EKS cluster, deploy Karpenter, deploy a sample application, and demonstrate node scaling with Karpenter. We will also look at the process of deploying constraints with pods in line to requirements of provisioner API for different application workloads or multiple teams needing different instance capacity for their application.
Prerequisites
Karpenter deployment tasks
Create an Amazon EKS cluster and node group. Then set up Karpenter and deploy Provisioner API.
- Set the following environment variables:
export CLUSTER_NAME=karpenter-demo
export AWS_DEFAULT_REGION=ap-south-1
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
- Create a cluster with eksctl. This example configuration file specifies a basic cluster with one initial node and sets up an IAM OIDC provider for the cluster to enable IAM roles for pods.
- Note: For an existing EKS cluster, you can determine whether you have one or need to create one in Create an IAM OIDC provider for your cluster.
eksctl create cluster -f - << EOF
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: ${CLUSTER_NAME}
region: ${AWS_DEFAULT_REGION}
version: "1.20"
managedNodeGroups:
- instanceType: m5.large
amiFamily: AmazonLinux2
name: ${CLUSTER_NAME}-ng
desiredCapacity: 1
minSize: 1
maxSize: 2
iam:
withOIDC: true
EOF
export CLUSTER_ENDPOINT="$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.endpoint" --output text)"
- Create subnet tags
kubernetes.io/cluster/$CLUSTER_NAME
.
- Karpenter discovers subnets tagged kubernetes.io/cluster/$CLUSTER_NAME. Add this tag to associated subnets of your cluster. Retrieve the subnet IDs and tag them with the cluster name.
SUBNET_IDS=$(aws cloudformation describe-stacks \
--stack-name eksctl-${CLUSTER_NAME}-cluster \
--query 'Stacks[].Outputs[?OutputKey==`SubnetsPrivate`].OutputValue' \
--output text)
aws ec2 create-tags \
--resources $(echo $SUBNET_IDS | tr ',' '\n') \
--tags Key="kubernetes.io/cluster/${CLUSTER_NAME}",Value=
- Create the KarpenterNode IAM role.
Instances launched by Karpenter must run with an InstanceProfile that grants permissions necessary to run containers and configure networking. Karpenter discovers the InstanceProfile using the name KarpenterNodeRole-${ClusterName}.
TEMPOUT=$(mktemp)
curl -fsSL https://karpenter.sh/v0.6.3/getting-started/cloudformation.yaml > $TEMPOUT \
&& aws cloudformation deploy \
--stack-name Karpenter-${CLUSTER_NAME} \
--template-file ${TEMPOUT} \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides ClusterName=${CLUSTER_NAME}
- Grant access to instances using the profile to connect to the cluster. Add the Karpenter node role to your aws-auth configmap.
eksctl create iamidentitymapping \
--username system:node:{{EC2PrivateDNSName}} \
--cluster ${CLUSTER_NAME} \
--arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME} \
--group system:bootstrappers \
--group system:nodes
- Create the KarpenterController IAM role. Karpenter requires permissions like launching instances. This will create an AWS IAM role and a Kubernetes service account and associate them using IRSA.
eksctl create iamserviceaccount \
--cluster "${CLUSTER_NAME}" --name karpenter --namespace karpenter \
--role-name "${CLUSTER_NAME}-karpenter" \
--attach-policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}" \
--role-only \
--approve
export KARPENTER_IAM_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"
- Install Karpenter Helm chart.
helm repo add karpenter https://charts.karpenter.sh
helm repo update
helm upgrade --install --namespace karpenter --create-namespace \
karpenter karpenter/karpenter \
--version v0.6.3 \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
--set clusterName=${CLUSTER_NAME} \
--set clusterEndpoint=${CLUSTER_ENDPOINT} \
--set aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
--wait # for the defaulting webhook to install before creating a Provisioner
- Enable debug logging (optional).
kubectl patch configmap config-logging -n karpenter --patch '{"data":{"loglevel.controller":"debug"}}'
- Deploy the provisioner and application pods with layered constraints by applying the following Karpenter provisioner spec. It has the requirements for architecture type (arm64 & amd64), capacity type (Spot & On-demand), and taints for GPU-based use cases.
cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
requirements:
- key: "karpenter.sh/capacity-type"
operator: In
values: ["spot", "on-demand"]
- key: "kubernetes.io/arch"
operator: In
values: ["arm64", "amd64"]
limits:
resources:
cpu: 1000
provider:
subnetSelector:
kubernetes.io/cluster/$CLUSTER_NAME: '*'
securityGroupSelector:
kubernetes.io/cluster/$CLUSTER_NAME: '*'
ttlSecondsAfterEmpty: 30
EOF
- Run the application deployment on a specific capacity, instance type, hardware, and Availability Zone using pod scheduling constraints.
Sample deployment
In the following sample deployment, we define the nodeSelector with topology.kubernetes.io/zone
to choose an Availability Zone and on-demand arm64 instance with karpenter.sh/capacity-type
and kubernetes.io/arch: arm64
and specific instance type node.kubernetes.io/instance-type
so that new nodes can be launched by Karpenter using the following pod scheduling constraints.
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: inflate
spec:
replicas: 0
selector:
matchLabels:
app: inflate
template:
metadata:
labels:
app: inflate
spec:
nodeSelector:
node.kubernetes.io/instance-type: r6gd.xlarge
karpenter.sh/capacity-type: on-demand
topology.kubernetes.io/zone: ap-south-1a
kubernetes.io/arch: arm64
terminationGracePeriodSeconds: 0
containers:
- name: inflate
image: public.ecr.aws/eks-distro/kubernetes/pause:3.5
resources:
requests:
cpu: 1
EOF
- Scale the above deployment to see the node scaling via Karpenter. It will choose the previous configuration from the Amazon EC2 fleet via the createFleet API for the application pods.
kubectl scale deployment inflate --replicas 3
- Review the Karpenter pod logs for events and more details.
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
- Example snippet of the logs:
eksadmin:~/environment $ kubectl logs -f -n karpenter -l app.kubernetes.io/name*=*karpenter -c controller
...
...
2022-02-18T10:48:15.913Z INFO controller.provisioning Batched 3 pods in 1.0555487s {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T10:48:15.918Z INFO controller.provisioning Computed packing of 1 node(s) for 3 pod(s) with instance type option(s) [r6gd.xlarge] {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T10:48:16.004Z DEBUG controller.provisioning Discovered kubernetes version 1.20 {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T10:48:16.042Z DEBUG controller.provisioning Discovered ami-06a248526462b970e for query /aws/service/eks/optimized-ami/1.20/amazon-linux-2-arm64/recommended/image_id {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T10:48:17.782Z INFO controller.provisioning Launched instance: i-034b68cf08d334acf, hostname: ip-192-168-135-148.ap-south-1.compute.internal, type: r6gd.xlarge, zone: ap-south-1a, capacityType: on-demand {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T10:48:17.821Z INFO controller.provisioning Bound 3 pod(s) to node ip-192-168-135-148.ap-south-1.compute.internal {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T10:48:17.821Z INFO controller.provisioning Waiting for unschedulable pods {"commit": "fd19ba2", "provisioner": "default"}
- Validate the application pods with the following command and the same will be in a Running state.
kubectl get node -L node.kubernetes.io/instance-type,kubernetes.io/arch,karpenter.sh/capacity-type
kubectl get pods -o
wide
Example snippet of the node output and pods output.
eksadmin:~/environment $ kubectl get node -L node.kubernetes.io/instance-type,kubernetes.io/arch,karpenter.sh/capacity-type
NAME STATUS ROLES AGE VERSION INSTANCE-TYPE ARCH CAPACITY-TYPE
ip-192-168-135-148.ap-south-1.compute.internal Ready <none> 79s v1.20.11-eks-f17b81 r6gd.xlarge arm64 on-demand
ip-192-168-4-178.ap-south-1.compute.internal Ready <none> 22m v1.20.11-eks-f17b81 m5.large amd64
eksadmin:~/environment $
eksadmin:~/environment $ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
inflate-56c45f4967-6x9h9 1/1 Running 0 87s 192.168.159.41 ip-192-168-135-148.ap-south-1.compute.internal <none> <none>
inflate-56c45f4967-cffdd 1/1 Running 0 87s 192.168.145.73 ip-192-168-135-148.ap-south-1.compute.internal <none> <none>
inflate-56c45f4967-xq4tw 1/1 Running 0 87s 192.168.130.79 ip-192-168-135-148.ap-south-1.compute.internal <none> <none>
eksadmin:~/environment $
We can see Karpenter applied layered constraints to launch nodes that satisfy multiple scheduling constraints of a workload, like instance type, specific Availability Zone, and hardware architecture via Karpenter.
Groupless Node upgrades
When using the node groups (self-managed or managed) with an EKS cluster, and as part of upgrading the worker nodes to a newer version of Kubernetes, we would have to rely on either migrating to a new node group for self-managed or launching a new autoscaling group of worker nodes for a managed node group, as mentioned in managed node group update behavior. Whereas with the Karpenter groupless autoscaling the upgrade of nodes works with the expiry time-to-live value.
Karpenter Provisioner API has Node Expiry that will allow a node to expire on reaching the expiry time-to-live value (ttlSecondsUntilExpired). The same value is used to upgrade nodes ttlSecondsUntilExpired
. The nodes will be terminated after a set period of time, after which they are replaced with newer nodes.
Note: Karpenter supports using custom launch templates. When using a custom launch template, you are taking responsibility for maintaining the launch template, including updating which AMI is used (that is, for security updates). In the default configuration, Karpenter will use the latest version of the EKS optimized AMI, which is maintained by AWS.
- Validate the current EKS cluster Kubernetes version with the following command.
aws eks describe-cluster --name ${CLUSTER_NAME} | grep -i version
The following is an example snippet of the previous command:
eksadmin:~/environment $ aws eks describe-cluster --name karpenter-demo | grep -i version
"platformVersion": "eks.3",
"version": "1.20",
eksadmin:~/environment $
- Deploy PodDisruptionBudget for your application deployment. PodDisruptionBudget (PDB) limits the number of pods of a replicated application that are down simultaneously from voluntary disruptions.
cat <<EOF | kubectl apply -f -
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: inflate-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: inflate
EOF
The following is an example snippet of previous PDB and sample application deployment that was configured in an earlier section:
eksadmin:~/environment $ kubectl get pdb
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
inflate-pdb 2 N/A 1 2m7s
eksadmin:~/environment $
eksadmin:~/environment $ kubectl get deploy inflate
NAME READY UP-TO-DATE AVAILABLE AGE
inflate 3/3 3 3 39m
eksadmin:~/environment $
- Upgrade the EKS cluster to a newer Kubernetes version via the console or eksctl as mentioned in the EKS documentation. We can see that the cluster was upgraded successfully to
1.21
.
eksadmin:~/environment $ aws eks describe-cluster --name ${CLUSTER_NAME} | grep -i version
"alpha.eksctl.io/eksctl-version": "0.79.0",
"platformVersion": "eks.4",
"version": "1.21",
eksadmin:~/environment $
- Checking our workload and node created by Karpenter earlier, we can see that nodes are of version
1.20
as Karpenter used the latest version of the EKS optimized AMI based on the earlier EKS cluster version 1.20
.
eksadmin:~/environment $ kubectl get node -L node.kubernetes.io/instance-type,kubernetes.io/arch,karpenter.sh/capacity-type
NAME STATUS ROLES AGE VERSION INSTANCE-TYPE ARCH CAPACITY-TYPE
ip-192-168-135-148.ap-south-1.compute.internal Ready <none> 115m v1.20.11-eks-f17b81 r6gd.xlarge arm64 on-demand
ip-192-168-4-178.ap-south-1.compute.internal Ready <none> 137m v1.20.11-eks-f17b81 m5.large amd64
eksadmin:~/environment $
eksadmin:~/environment $ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
inflate-56c45f4967-6x9h9 1/1 Running 0 116m 192.168.159.41 ip-192-168-135-148.ap-south-1.compute.internal <none> <none>
inflate-56c45f4967-cffdd 1/1 Running 0 116m 192.168.145.73 ip-192-168-135-148.ap-south-1.compute.internal <none> <none>
inflate-56c45f4967-xq4tw 1/1 Running 0 116m 192.168.130.79 ip-192-168-135-148.ap-south-1.compute.internal <none> <none>
eksadmin:~/environment $
- Now, let’s reconfigure the provisioner API of Karpenter and append
ttlSecondsUntilExpired
. This will add the node expiry, which allows the nodes to get terminated and replaced with a new one matching the EKS cluster Kubernetes version 1.21
now.
cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
requirements:
- key: "karpenter.sh/capacity-type"
operator: In
values: ["spot", "on-demand"]
- key: "kubernetes.io/arch"
operator: In
values: ["arm64", "amd64"]
limits:
resources:
cpu: 1000
provider:
subnetSelector:
kubernetes.io/cluster/$CLUSTER_NAME: '*'
securityGroupSelector:
kubernetes.io/cluster/$CLUSTER_NAME: '*'
ttlSecondsAfterEmpty: 30
ttlSecondsUntilExpired: 1800
EOF
Note: If ttlSecondsUntilExpired is nil, that means that the feature is disabled and nodes will never expire. For an example value, we can configure the node expiry to a value of 30 days as ttlSecondsUntilExpired: 2592000
(# 30 Days = 60 * 60 * 24 * 30 Seconds).
- Review the Karpenter pod logs for events and more details.
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
The following is an example snippet of the logs:
2022-02-18T12:48:31.289Z DEBUG controller.provisioning Discovered kubernetes version 1.21 {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T12:48:31.289Z DEBUG controller.provisioning Discovered ami-0aa1d81ef621af702 for query /aws/service/eks/optimized-ami/1.21/amazon-linux-2-arm64/recommended/image_id {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T12:48:31.414Z DEBUG controller.provisioning Created launch template, Karpenter-karpenter-demo-7876638028496176806 {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T12:48:32.732Z DEBUG controller.eviction Did not evict pod default/inflate-56c45f4967-6x9h9 due to PDB violation. {"commit": "fd19ba2"}
2022-02-18T12:48:32.740Z DEBUG controller.eviction Did not evict pod default/inflate-56c45f4967-xq4tw due to PDB violation. {"commit": "fd19ba2"}
2022-02-18T12:48:33.089Z INFO controller.provisioning Launched instance: i-0a3b7cb681485f416, hostname: ip-192-168-132-35.ap-south-1.compute.internal, type: r6gd.xlarge, zone: ap-south-1a, capacityType: on-demand {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T12:48:33.110Z INFO controller.provisioning Bound 1 pod(s) to node ip-192-168-132-35.ap-south-1.compute.internal {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T12:48:33.110Z INFO controller.provisioning Waiting for unschedulable pods {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T12:48:35.944Z DEBUG controller.eviction Did not evict pod default/inflate-56c45f4967-6x9h9 due to PDB violation. {"commit": "fd19ba2"}
2022-02-18T12:48:35.952Z DEBUG controller.eviction Did not evict pod default/inflate-56c45f4967-xq4tw due to PDB violation. {"commit": "fd19ba2"}
2022-02-18T12:48:42.354Z DEBUG controller.eviction Did not evict pod default/inflate-56c45f4967-6x9h9 due to PDB violation. {"commit": "fd19ba2"}
2022-02-18T12:49:42.451Z DEBUG controller.eviction Evicted pod default/inflate-56c45f4967-6x9h9 {"commit": "fd19ba2"}
2022-02-18T12:49:42.460Z INFO controller.provisioning Bound 1 pod(s) to node ip-192-168-132-35.ap-south-1.compute.internal {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T12:49:42.467Z DEBUG controller.eviction Did not to evict pod default/inflate-56c45f4967-xq4tw due to PDB violation. {"commit": "fd19ba2"}
2022-02-18T12:49:52.510Z DEBUG controller.eviction Evicted pod default/inflate-56c45f4967-xq4tw {"commit": "fd19ba2"}
2022-02-18T12:49:57.747Z INFO controller.termination Deleted node {"commit": "fd19ba2", "node": "ip-192-168-135-148.ap-south-1.compute.internal"}
2022-02-18T12:49:57.765Z INFO controller.provisioning Bound 1 pod(s) to node ip-192-168-132-35.ap-south-1.compute.internal {"commit": "fd19ba2", "provisioner": "default"}
2022-02-18T12:49:57.888Z INFO controller.provisioning Waiting for unschedulable pods {"commit": "fd19ba2", "provisioner": "default"}
Note: In the previous logs, we can see that PodDisruptionBudget was respected by Karpenter, and then it Discovered kubernetes version 1.21
, used the latest version of the EKS optimized AMI for 1.21
, and launched a new node for the workload. Later, the old node was cordoned, drained, and deleted by Karpenter.
If we validate the application pods with the following commands, we can see that Karpenter launched nodes are upgraded to 1.21
, same as that of the EKS cluster Kubernetes version.
kubectl get node -L node.kubernetes.io/instance-type,kubernetes.io/arch,karpenter.sh/capacity-type
kubectl get pods -o wide
The following is an example snippet of the node output and pods output:
eksadmin:~/environment $ kubectl get node -L node.kubernetes.io/instance-type,kubernetes.io/arch,karpenter.sh/capacity-type
NAME STATUS ROLES AGE VERSION INSTANCE-TYPE ARCH CAPACITY-TYPE
ip-192-168-132-35.ap-south-1.compute.internal Ready <none> 113s v1.21.5-eks-9017834 r6gd.xlarge arm64 on-demand
ip-192-168-4-178.ap-south-1.compute.internal Ready <none> 143m v1.20.11-eks-f17b81 m5.large amd64
eksadmin:~/environment $
eksadmin:~/environment $ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
inflate-56c45f4967-4cdx6 1/1 Running 0 49s 192.168.129.193 ip-192-168-132-35.ap-south-1.compute.internal <none> <none>
inflate-56c45f4967-gzgbh 1/1 Running 0 39s 192.168.158.104 ip-192-168-132-35.ap-south-1.compute.internal <none> <none>
inflate-56c45f4967-hfqwb 1/1 Running 0 2m2s 192.168.152.242 ip-192-168-132-35.ap-south-1.compute.internal <none> <none>
eksadmin:~/environment $
In the previous demonstration, we see that Karpenter respected the PDB and its ability to apply node expiry for upgrading of nodes launched by Karpenter.
Node expiry can be used as a means of upgrading or repacking nodes so that nodes are retired and replaced with updated versions. See How Karpenter nodes are deprovisioned in the Karpenter documentation for information on using ttlSecondsUntilExpired and ttlSecondsAfterEmpty.
Cleanup
Delete all the provisioners (CRDs) that were created.
kubectl delete provisioner default
Remove Karpenter and delete the infrastructure from your AWS account.
helm uninstall karpenter --namespace karpenter
eksctl delete iamserviceaccount --cluster ${CLUSTER_NAME} --name karpenter --namespace karpenter
aws cloudformation delete-stack --stack-name Karpenter-${CLUSTER_NAME}
aws ec2 describe-launch-templates \
| jq -r ".LaunchTemplates[].LaunchTemplateName" \
| grep -i Karpenter-${CLUSTER_NAME} \
| xargs -I{} aws ec2 delete-launch-template --launch-template-name {}
eksctl delete cluster --name ${CLUSTER_NAME}
Conclusion
Karpenter provides the option to scale nodes quickly and with very little latency. In this blog, we demonstrated how the nodes can be scaled with different options for each use case using Provisioner API by leveraging the well-known Kubernetes labels and taints and using the pod scheduling constraints within the deployment so that Pods get deployed on the Karpenter provisioned nodes. This demonstrates that we can run different types of workloads on different capacities or requirements for each of its use cases. Further, we see the upgrade node behavior for the nodes launched by Karpenter by enabling the node expiry time ttlSecondsUntilExpired
with the provisioner API.