Skip to content

czam01/mwaa

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

78 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

#Managed Apache Airflow on AWS

Deployment on MWAA

Bucket for Plugins/DAGs/Requirments

  • You need to specify a bucket that must start with airlow in the name.
  • Specify different paths for Dags, Plugins and Requirements.

Custom Configurations

  • Create a new VPC and select the type of deployment (public or private).
  • Select the environment class, to start I suggest mw1.small.
  • For monitoring in production environment it is better to enable form INFO logs for tasks, webserver, scheduler, worker and DAGs.
  • Create a custom role and give the role permissions over the S3 bucket. (Remember give permissions to the rone in the Bucket policy).

DAG Structure

  • Store all the DAGs in the dags folder.
  • The DAGs should finish with .py extension.

Lesson Learned

  • Take into account that the latest version is 1.10.12 and you will have all the default packages that come with the version.
  • In the requirements file you need to specify additional packages in this way: apache-airflow[slack, package2, package3].
  • If you need to use custom operators you need to call them from the dag file: from airflow.contrib.operators.slack_webhook_operator import SlackWebhookOperator

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages