This repository provides a deployable example for the blog post "Running Celery workers using AWS Batch" blog post, which discusses how to create an architecture for deploying Celery workers using AWS Batch. The following figure (Figure 1) contains a high-level architecture of the example:

In the diagram:

1. A Celery application submits tasks to an SQS queue
2. Amazon CloudWatch alarms monitor the depth of the queue, and enter ALARM state when the approximate visible message depth is >=5 and >=50.
3. Amazon Event Bridge rules react to those alarms to start the Celery workers with AWS Batch. The worker processes drain the SQS queue and shut themselves down when the queue is empty for more than 60 seconds.

You will find the example Celery Python code in the cdk-project/batch_celery_container directory.

• app.py - The Celery app that defines the task to send to the SQS queue and for the Celery worker launched by AWS Batch to process.
• fill-batch-queue.py - A small program that imports the Celery app and actually submits a number of Celery requests into the queue for processing.

This example leverages the AWS CDK for Python to define and manage the infrastructure, including:

• A VPC with a public and private subnet deployed to a single Availability Zone. The private subnet has a NAT gateway attached.
• A Docker container and private ECR repository for the Celery processes.
• An AWS Batch compute environment that leverages Fargate to deploy resources to the private VPC.
• An AWS Batch FIFO job queue attached to the compute environment.
• AWS Batch job definitions that reference the Celery container for submitting the Celery tasks for Batch and for running Celery workers in Batch
• An SQS queue for Celery tasks meant for Batch
• Amazon CloudWatch alarms set to enter ALARM state when the SQS approximate queue depth reaches >= 5 and >= 50 messages
• Amazon EventBridge rules that react to the alarms to submit jobs to AWS Batch to run Celery workers.

If you would like to deploy and run this example in your own AWS account, please refer to the README.md file within the cdk-project subdirectory.

See CONTRIBUTING for more information.

This library is licensed under the MIT-0 License. See the LICENSE file.