This repository demonstrates building a Continuous Integration and Continuous Delivery (CI/CD) pipeline using AWS CodeCommit, AWS CodeBuild and AWS CodePipeline to provision AWS Glue based data pipelines using AWS Cloud Development Kit (CDK).

Please refer to the accompanying post Build, Test and Deploy ETL solutions using AWS Glue and AWS CDK based CI/CD pipelines for additional information.

The AWS CDK based solution deploys a CI/CD pipeline using AWS CodeCommit, AWS CodeBuild and AWS CodePipeline, executes docker based PyTests and deploys a data pipeline using AWS Glue. The data pipeline consists of AWS Glue Workflow, Triggers, Jobs and Crawlers. The data pipeline uses COVID-19 Harmonized Data managed by Talend / Stitch to construct a data pipeline.

The diagram shows the various components of the solution:

• An AWS account
• The AWS Command Line Interface (AWS CLI) installed
• The GIT Command Line Interface (GIT CLI) installed
• The AWS CDK Toolkit (cdk command) installed
• Python 3 installed
• Permissions to create AWS resources

Please follow the below instructions to deploy the solution in your AWS account:

$ git clone https://github.com/aws-samples/aws-glue-cdk-cicd.git

$ cd aws-glue-cdk-cicd
$ python3 -m venv .venv

$ source .venv/bin/activate

% .venv\Scripts\activate.bat

$ pip install -r requirements.txt

$ cdk bootstrap 

The above command will create the below CloudFormation stack:

• CDKToolkit.

$ cdk synth  # optional if not default (-c stage=default)

$ cdk deploy  # optional if not default (-c stage=default)

The above command will create the below CloudFormation stack:

• cdk-covid19-glue-stack-pipeline
• cdk-covid19-glue-stack

$ aws glue start-workflow-run --name Covid_19

SELECT "country", "location", "date", "cases", "deaths", "ecdc-countries", "acute_care", "acute_care_per_100K", "critical_care", "critical_care_per_100K" FROM "AwsDataCatalog"."covid19db"."neherlab_denormalized" limit 10;

Log into the AWS console, select the appropriate region, navigate to the AWS CloudFormation console , and select and click Delete on the following stacks:

• cdk-covid19-glue-stack-pipeline
• cdk-covid19-glue-stack
• CDKToolkit.

Log into the AWS console, select the appropriate region, navigate to the AWS S3 console and delete the associated S3 buckets:

• cdk-covid19-glue-stack-p-pipelineartifactsbucketa-*
• cdk-*-assets-<AWS_ACCOUNT_ID>-<AWS_REGION>
• covid19-glue-config-<AWS_ACCOUNT_ID>-<AWS_REGION>
• neherlab-denormalized-dataset-<AWS_ACCOUNT_ID>-<AWS_REGION>

See CONTRIBUTING for more information.

This library is licensed under the MIT-0 License. See the LICENSE file.