Building an Amazon EKS Cluster Preconfigured to Run High Traffic Microservices

The modern digital landscape thrives on high traffic platforms, such as bustling online marketplaces, media streaming services, and real-time data analytics applications. These platforms often face unpredictable surges in user activity, like a sudden influx of ticket sales due to a concert announcement or a streaming service being swamped during a much-anticipated series premiere. Such scenarios require an architecture that can dynamically scale and swiftly recover from failures. Microservices are key to meeting these demands, but the real challenge lies in their efficient orchestration. Amazon EKS stands out as a solution, offering streamlined deployment and scaling of services, allowing each one to independently handle spikes without affecting others. With the robust support of AWS's infrastructure, Amazon EKS ensures both performance and reliability.

This tutorial shows you how to create a managed node groups-based Amazon EKS cluster using an eksctl “quickstart” template. This use case-specific template creates and sets up a cluster preconfigured and ready to run your dynamic frontends, such as interactive web dashboards, and data-intensive backends, such as analytics engines or recommendation systems.

About
✅ AWS experience	200 - Intermediate
⏱ Time to complete	30 minutes
🧩 Prerequisites	- AWS Account
📢 Feedback	Any feedback, issues, or just a 👍 / 👎 ?
⏰ Last Updated	2023-08-29

Prerequisites

Before you begin this tutorial, you need to:

Install the latest version of kubectl. To check your version, run: kubectl version --short.
Install the latest version of eksctl. To check your version, run: eksctl info.

Overview

This tutorial is the first part of a series on managing high traffic microservices platforms using Amazon EKS, and it's dedicated to preconfiguring a cluster with the components it needs to run microservice applications with data-intensive workloads. Using the eksctl cluster template that follows, you'll build a robust, scalable, and secure Amazon EKS cluster with managed node groups. This template not only enables application workloads but also fortifies the cluster with an additional layer of security, fully aligned with best practices for production environments. It configures the following components:

Autoscaling: Managed node groups use an m5.large instance type, providing a balance of resources. With a minimum size of "2" and a maximum size of "5", node groups can dynamically scale. The volume size is set to "100", ensuring ample capacity, and required subnet tags allow the Kubernetes Cluster Autoscaler (CA) to dynamically scale your cluster.
Authentication: Necessary IAM Roles for Service Accounts (IRSAs) mappings to enable communication between Kubernetes pods and AWS services. This includes the AWS Load Balancer Controller (LBC) used to expose applications, Amazon EFS CSI Driver for persistent data storage, Kubernetes External DNS to automatically manage DNS records, and Cert Manager to streamline management of SSL/TLS certificates. Additionally, an OpenID Connect (OIDC) endpoint enables seamless and secure communication.
Add-ons: Latest versions of the following add-ons, including "vpc-cni" to enable the Amazon VPC Container Network Interface, "coredns" to facilitate DNS resolution, "kube-proxy" to maintain network rules on each Amazon EC2 node, and the EBS CSI Driver Add-On.
Public/Private Networking: Managed node groups utilize private networking and a NAT gateway to bolster security by limiting direct internet access. The AWS Load Balancer Controller (LBC) manages and securely distributes all incoming web traffic to private subnets.
Monitoring: An Amazon CloudWatch IAM policy is attached to the IAM Role for Service Account (IRSA), aiding optional components like CloudWatch Container Insights to collect and summarize metrics and logs.

Note that if you're still within your initial 12-month AWS Free Tier period, certain Amazon EC2 instances for managed node groups and additional AWS services may not be included in the Free Tier, and charges may apply based on your usage.

Step 1: Configure the Cluster

In this section, you will configure the Amazon EKS cluster to meet the specific demands of high-traffic microservice applications. By creating this cluster-config.yaml file, you'll define the settings for IAM roles, scalable resources, private networking, and monitoring. These configurations are essential for ensuring that the cluster is robust, scalable, and secure, with optimized performance for dynamic scalability and data persistence.

To Create the Cluster Config

Create a cluster-config.yaml file and paste the following contents into it. Replace the region with your preferred region.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: managednodes-quickstart
  region: us-east-2
  version: "1.27"
  tags:
    # Add more cloud tags if needed for billing
    environment: managednodes-quickstarts

# The IAM section is for managing IAM roles and service accounts for your cluster.
iam:
  withOIDC: true
  serviceAccounts:
  - metadata:
  # Service account used by Container Insights. (See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-prerequisites.html)  
      name: cloudwatch-agent
      namespace: amazon-cloudwatch
      labels: {aws-usage: "container-insights"}
    attachPolicyARNs:
    - "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"    
  - metadata:
      # Service account used by the AWS Load Balancer Controller.
      name: aws-load-balancer-controller
      namespace: kube-system
    wellKnownPolicies:
      awsLoadBalancerController: true    
  - metadata:
  # Service account used by Amazon EFS CSI driver. (See https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html)
      name: efs-csi-controller-sa
      namespace: kube-system
    wellKnownPolicies:
      efsCSIController: true
  - metadata:
  # Service account used by External DNS. (See https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.5/guide/integrations/external_dns/)
      name: external-dns
      namespace: kube-system
    wellKnownPolicies:
      externalDNS: true
  - metadata:
  # Service account used by Certificate Manager. (See https://cert-manager.io/docs/installation/)
      name: cert-manager
      namespace: cert-manager
    wellKnownPolicies:
      certManager: true   
  - metadata:
  # Service account used by Cluster Auto-scaler. (See https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/CA_with_AWS_IAM_OIDC.md)
      name: cluster-autoscaler
      namespace: kube-system
      labels: {aws-usage: "autoscaling-worker-nodes"}
    wellKnownPolicies:
      autoScaler: true

managedNodeGroups:
  - name: managed-ng
    instanceType: m5.large
    minSize: 2
    desiredCapacity: 3
    maxSize: 5
    # launch nodegroup in private subnets
    privateNetworking: true
    volumeSize: 100
    volumeType: gp3
    # Encrypt Worker Nodes Amazon EBS Volumes by default
    volumeEncrypted: true
    labels: 
      node-class: "production-workload" 
      role: "worker"   
    tags:
      nodegroup-role: worker
      env: prod     
      # EC2 tags required for cluster-autoscaler auto-discovery - these tags are automatically applied to a managed nodegroup autoscaling group
      k8s.io/cluster-autoscaler/enabled: "true"
      k8s.io/cluster-autoscaler/managednodes-quickstart: "owned"      

addons:
  - name: vpc-cni # no version is specified so it deploys the default version
    attachPolicyARNs:
      - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
  - name: coredns
    version: latest # auto discovers the latest available
  - name: kube-proxy
    version: latest
  - name: aws-ebs-csi-driver
    wellKnownPolicies:      # add IAM and service account
      ebsCSIController: true

cloudWatch:
 clusterLogging:
   enableTypes: ["*"]
    # Sets the number of days to retain the logs for (see [CloudWatch docs](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutRetentionPolicy.html#API_PutRetentionPolicy_RequestSyntax)).
    # By default, log data is stored in CloudWatch Logs indefinitely.
   logRetentionInDays: 60

Step 2: Create the Cluster

Now, we're ready to create our Amazon EKS cluster. This process takes several minutes to complete. If you'd like to monitor the status, see the AWS CloudFormation console.

Create the EKS cluster using the cluster-config.yaml.

1
eksctl create cluster -f cluster-config.yaml

Upon completion, you should see the following response output:

1
2023-08-26 08:59:54 [✔]  EKS cluster "managednodes-quickstart" in "us-east-2" region is ready

When the previous command completes, verify that all of your nodes have reached the Ready state with the following command:

1
kubectl get nodes

The expected output should look like this:

1
2
3
4
NAME                                            STATUS   ROLES    AGE   VERSION
ip-192-168-119-7.us-east-2.compute.internal     Ready    <none>   27m   v1.27.4-eks-8ccc7ba
ip-192-168-141-57.us-east-2.compute.internal    Ready    <none>   19m   v1.27.4-eks-8ccc7ba
ip-192-168-177-109.us-east-2.compute.internal   Ready    <none>   19m   v1.27.4-eks-8ccc7ba

Step 3: Verify Cluster Node and Pod Health

Verify that the Amazon EBS CSI Driver was successfully installed with the following command:

1
kubectl get deployment ebs-csi-controller -n kube-system

The expected output should look like this:

1
2
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
ebs-csi-controller   2/2     2            2           15m

Verify all the service accounts created in the cluster with the following command:

1
 kubectl get sa -A | egrep "cert-manager|efs|aws-load|external|cloudwatch-agent|cluster-autoscaler"

The expected output should look like this:

1
2
3
4
5
6
7
amazon-cloudwatch   cloudwatch-agent                     0         43m
cert-manager        cert-manager                         0         43m
cert-manager        default                              0         43m
kube-system         aws-load-balancer-controller         0         42m
kube-system         cluster-autoscaler                   0         43m
kube-system         efs-csi-controller-sa                0         43m
kube-system         external-dns                         0         43m

Congratulations! You've successfully set up the foundational infrastructure of your Amazon EKS cluster, laying the essential groundwork for deploying high-traffic microservices workloads. This setup provides you with the groundwork needed to begin deploying applications, but keep in mind that additional configurations are required for a fully optimized, production-ready environment.

(Optional) Deploy a Sample Application

Now, we’re all set to launch a sample application and enable its accessibility on the internet through an Application Load Balancer (ALB). For step-by-step guidance, check out the tutorial at Exposing and Grouping Applications using the AWS Load Balancer Controller (LBC) on an EKS IPv4 Cluster. This tutorial will guide you through the required Ingress annotations for the AWS LBC, an essential mechanism for controlling external access to services within an EKS cluster. You’ll also explore Ingress Groupings, a sophisticated feature that amalgamates multiple Ingress resources into one ALB, enhancing both efficiency and ALB management.

Clean Up

To avoid incurring future charges, you should delete the resources created during this tutorial. You can delete the EKS cluster with the following command:

1
eksctl delete cluster -f ./cluster-config.yaml

Upon completion, you should see the following response output:

1
2023-08-26 17:26:44 [✔]  all cluster resources were deleted

Conclusion

In this tutorial, you've not only successfully set up an Amazon EKS cluster optimized for deploying microservice applications but also laid the foundation for a seamless integration between AWS and Kubernetes. By configuring the necessary IAM Roles for Service Accounts (IRSAs), node groups, and other essential components, you've established the infrastructure that ensures smooth communication with AWS services and drivers needed to run high-traffic microservices. To fully leverage your new setup, remember that the installation of the ExternalDNS, Cluster Autoscaler, Container Insights, and AWS Load Balancer Controller are still required. With these final installations, you'll have a robust, fully operational environment ready for your microservice application deployment, all while taking advantage of the unique strengths of both Amazon EKS and Kubernetes.

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Site Terms, Privacy, and more.

Building an Amazon EKS Cluster Preconfigured to Run High Traffic Microservices

Deploy a preconfigured Amazon EKS cluster optimized for high-demand microservice applications using an eksctl "quickstart" template.

Prerequisites

Overview

Step 1: Configure the Cluster

To Create the Cluster Config

Step 2: Create the Cluster

Step 3: Verify Cluster Node and Pod Health

(Optional) Deploy a Sample Application

Clean Up

Conclusion

Comments