This is a sample implementation of Azure Event Hub consumer on ECS, storing checkpoint and ownership information in a DynamoDB table, and saving all the event data to a S3 bucket. Additionaly it reports metrics for consumer latency to CloudWatch Metrics. You can further use this metrics for autoscaling ECS service, monitoring how well consumers are working, and more.

Below is the architecture diagram for this sample.

Before deploying this sample, you must install AWS Cloud Development Kit prerequisites. Please refer to this document for the detailed instruction. Please make sure you've successfully completed cdk bootstrap step.

After that, clone this repository and go to infrastructure directory.

cd infrastructure

You must first install Node.js dependencies for CDK code by the following commands:

npm ci

Now you can deploy this sample stack (EventHubConsumerStack) by the following command:

CONNECTION_STRING=<your_eventhub_connection_string> npx cdk deploy --require-approval never

You must pass your Event Hub's connection string by CONNECTION_STRING environment variable. Please refer to Get an Event Hubs connection string to obtain it. Please note that your connection string must include the name of your Event Hub topic.

Initial deployment usually takes about 5 minutes.

After a successful deployment, you can check the ARN of the S3 bucket, in which all the events data will be saved.

When you send events to an Event Hub topic you specifid in the connection string, the events will be saved to the S3 bucket.

You can see saved event data in S3 management console. You can also check the consumer's execution logs in ECS management console, or metrics for consumer's latency in CloudWatch Metrics management console.

To avoid incurring future charges, clean up the resources you created.

You can remove all the AWS resources deployed by this sample running the following command:

cd infrastructure
npx cdk destroy --force

See CONTRIBUTING for more information.

This library is licensed under the MIT-0 License. See the LICENSE file.