Happy to lead this effort

Thank you @john-jac! Please let me know if you need help on the kubernetes or airflow core side of things. I think this would be a great addition to Airflow's k8s story :).

Hi folks, I'm going to be working with @john-jac on this; the code etc will be in my fork and I'll have a design plan/doc posted here in this ticket in the next day or two.

I've started putting down the boilerplate code, I think this is how I am going to tackle this. I'm not sure how formal you folks like it yet, but please let me know what you think of the plan.

Goals

Add a collection of Apache Airflow Operators which interact with Amazon Elastic Kubernetes Service (EKS) and abstract all Amazon logic from the Airflow user. This will allow Airflow to be a general purpose Kubernetes orchestrator able to do multi-cluster orchestration across multiple clouds.

Proposal

The proposed solution is a collection of Operators, and their underlying Hooks, which will be added to the Amazon AWS provider package. These Operators will handle creating and deleting clusters, as well as executing tasks using EKS Managed Node Groups.

Assumptions and Prerequisites

• The account running the DAGs will need eks:DescribeCluster IAM permissions to retrieve the information currently provided by the manual kubeconfig file.

Definitions

Pod - A Kubernetes pod is the way that Kubernetes runs containers on a compute instance and includes containers and specifications for how they should run, networking, and storage. A pod can be a single container or multiple containers that always run together.

Cluster - An Amazon EKS cluster consists of the Amazon EKS control plane, which runs the Kubernetes software and API server, and the pod that is registered with the control plane.

Operator - An operator defines a single task within the workflow.

kubectl - The Kubernetes command-line tool which allows users to run commands against Kubernetes clusters. Uses include deploying applications, inspecting and managing cluster resources, and viewing logs.

eksctl - An open source CLI tool created by the community to create clusters on EKS using CloudFormation.

aws eks (cli tool) - A CLI tool which, among other things, is used to generate the kubeconfig file.

kubeconfig - A config file containing required information about clusters, users, namespaces, and authentication mechanisms. kubectl uses kubeconfig files to find the information it needs to choose a cluster and communicate with the API server of a cluster.

EKS Managed Node Groups (nodegroup) - Infrastructure as a Service - EKS Managed Node Groups create and manage Amazon Elastic Compute Cloud (EC2) instances which host a Kubernetes cluster. This is the default underlying compute platform for EKS clusters.

Task - The process or command being run in a pod.

Context and User Experience

While the basic functions of creating and running pods on EKS can be handled through the existing Cloud Native Computing Foundation (CNCF) Kubernetes Pod Operator, running the pods on EKS introduces pain points to the users, some of which are detailed below, and requires some specific EKS knowledge. By abstracting away some of this Amazon-specific logic, we can automate and streamline the configuration and deployment of new pods.

Currently, in order to deploy a new pod on EKS, the user needs to leverage the kubectl, eksctl, and aws command-line tools and generate config files to manually pass data to the Kubernetes Pod Operator. The current manual process is:

1. Create a cluster - uses the eksctl CLI tool
2. Create a namespace - uses the kubectl CLI tool
3. Create and attach an IAM Role for permission to log into the cluster - uses eksctl CLI tool
4. Create or modify the Airflow requirements.txt file to ensure it contains two required packages: awscli and kubernetes==12.0.1
5. Create and possibly edit the kubeconfig file - uses aws eks CLI tool
6. Copy the edited kubeconfig file to the dags directory

Using the BOTO3 python API, new Operators can automate most or all of those steps and create a more seamless experience for the user.

Use Cases

Benchmarks

At a minimum, this solution should offer feature parity with the Google Kubernetes Engine (GKE) Pod Operator functionality.

Create Cluster - Create a Google Kubernetes Engine Cluster of specified dimensions

operator = GKEClusterCreateOperator(
                task_id='cluster_create',
                project_id='my-project',
                location='my-location',
                body=cluster_def,)

Delete Cluster - Deletes the cluster, including the Kubernetes endpoint and all worker nodes

operator = GKEClusterDeleteOperator(
                task_id='cluster_delete',
                project_id='my-project',
                location='cluster-location',
                name='cluster-name',)

Start Pod - Executes a task in a Kubernetes pod in the specified Google Kubernetes Engine cluster

 operator = GKEStartPodOperator(
                task_id="pod_task",
                project_id=GCP_PROJECT_ID,
                location=GCP_LOCATION,
                cluster_name=CLUSTER_NAME,
                namespace="default",
                image="perl",
                name="test-pod",)

And some sequence diagrams for the Operators for folks who like that kinda thing:

[EDIT: corrected CreateCluster sequence diagram image; initially uploaded the wrong file]

Hi @ferruzzi, this looks great!

In general, we don't need anything super formal for creating operators (though this is useful). If you were to make a core change that would involve submitting an AIP, but for this you're good to go!

Hi there, will this effort fix the reported problems with receiving logs from EKS pods in MWAA? Not sure if it's not just a general issue with Kubernetes logging mind you, but I had it confirmed by the AWS support team.

Either way, I've made some commits to the EMR operators in the past and would be happy to help out here.

Best I can tell that is an underlying issue which I don't intend to tackle at this time.

Quick update, most of the code for this has been implemented, but moto didn't have any EKS support. So I have a PR in with them to add that, then I'll drop a PR here once they push that to their release.