Integration & Automation

Use Git pre-commit hooks to avoid AWS CloudFormation errors

Do you struggle with formatting AWS CloudFormation templates? Do you wonder if your Python and shell scripts are 100% neat and error-free? Do you spend time and mental energy fixing missing semicolons and trailing white space at the end of your files? If so, there’s a better way to avoid errors: Use GitHub pre-commit hooks.

These hooks catch and sometimes even fix formatting issues and messy scripts before you commit changes to your Git repository.

With the approach we propose, you configure pre-commit hooks for your project and commit your changes to the repo. Then Git inspects your files and returns a list of any issues that must be fixed. If Git flags issues, the commit fails. If not, your code is committed to the project.

For example, here on the AWS Integration and Automation team, when we build AWS Quick Starts, we commonly use the following pre-commit hooks:

  • cfn-lint: AWS CloudFormation linter. Validates general mappings, resources, parameters, and other components in both YAML and JSON-based CloudFormation templates.
  • black: Formats your Python code.
  • isort: Sorts your imports in alphabetical order.
  • seed-isort-config: Sorts your third-party imports and integrates them into isort.
  • flake8: Enforces style consistency in Python code.
  • shellcheck: Checks shell scripts for accuracy.
  • mixed-line-ending: Replaces or checks mixed line endings.
  • trailing-whitespace: Trims trailing white space.

In this post, we walk you through an exercise that helps you learn, first-hand, how pre-commit hooks work.

About this blog post
Time to read ~15 min.
Time to complete ~25 min.
Cost to complete ~$0
Learning level Intermediate (200)
AWS services AWS CloudFormation
Amazon Elastic Kubernetes Service (Amazon EKS)

Process overview

Our exercise has five main steps:

  1. Fork and then clone the test repo.
  2. Install Python, pip, and the AWS CloudFormation linter hook.
  3. Configure your system and Git project for the pre-commit hooks.
  4. Install the pre-commit hooks.
  5. Test your pre-commit hooks.

For the test repo, we chose the Amazon EKS Quick Start because it represents complex configurations in multiple use cases. This Quick Start uses an AWS CloudFormation template to automate the deployment of the reference architecture shown in Figure 1. This architecture includes an Amazon EKS cluster, Kubernetes nodes spanning two or three Availability Zones, and a Linux bastion host in a public subnet.

Figure 1. Architecture diagram for the test repo, the Amazon EKS Quick Start

Prerequisites

This blog post assumes that you have a basic understanding of the following:

Walkthrough

You start this exercise by forking the Amazon EKS Quick Start in your GitHub account and then cloning it to your local machine, as described in step 1. When you work with a clone, you don’t accidentally push unwanted changes upstream to the active repo that’s in production.

Step 1: Fork and then clone the test repo

  1. Log in to your GitHub account at https://github.com/login.
  2. Go to https://github.com/aws-quickstart/quickstart-amazon-eks.
  3. In the upper right corner of the page, choose Fork.
  4. Choose your own account as the location of the fork.
  5. Clone the repo to a local directory on your system by running the following command. Replace <your-account> with the name of your account.
    git clone https://github.com/<your-account>/quickstart-amazon-eks.git

Step 2: Install Python, pip, and the AWS CloudFormation linter hook

To use pre-commit hooks, you need both Python and pip. (Python 3 is the programming language that we use to install the pre-commit hooks. Pip 3 is the standard package manager used with Python to manage additional libraries of code.) You need the AWS CloudFormation linter to run validation tests of your CloudFormation template files.

  1. Install Python 3 on your system. Follow the instructions in the Python 3 Installation & Setup Guide. For specific Windows releases of Python 3, see Python Releases for Windows.
  2. Install pip 3 on your system. Follow the instructions in Installing pip/setuptools/wheel with Linux Package Managers.
  3. Install the AWS CloudFormation linter hook by running this command: pip install cfn-lint

Step 3: Configure your system and Git project for the pre-commit hooks

In this section, you use pip to install the pre-commit hooks: isort (including the script-must-have-extension hook), seed-isort-config, black, flake8, mixed-line-editing, trailing-whitespace, and shellcheck. If you’re working with a non-Windows operating system, such as macOS, see the pre-commit documentation site for additional information.

  1. Install the pre-commit package manager by running this command: pip install pre-commit
  2. Navigate to the root directory of the Quick Start project that you forked earlier, and create the .pre-commit-config.yaml file by running the following command: touch .pre-commit-config.yaml
  3. Copy then paste the following text into the .pre-commit-config.yaml file. This step lets the installer know which pre-commit hooks to include in your project. Save your file.
    repos:
     -   repo: https://github.com/pre-commit/pre-commit-hooks
         rev: v2.3.0
         hooks:
         -   id: mixed-line-ending
         -   id: trailing-whitespace
         -   id: flake8
     -   repo: https://github.com/ambv/black
         rev: stable
         hooks:
         - id: black
           language_version: python3.7
     -   repo: https://github.com/asottile/seed-isort-config
         rev: v2.1.1
         hooks:
         -   id: seed-isort-config
             args: [--exclude=templates/]
     -   repo: https://github.com/pre-commit/mirrors-isort
         rev: v4.3.21  # Use the revision sha / tag you want to point at
         hooks:
         -   id: isort
             files: ^source/
             types: [file, python]
     -   repo: https://github.com/jumanjihouse/pre-commit-hooks
         rev: 2.0.0
         hooks:
         - id: script-must-have-extension
           name: QuickStart policy is to use .sh extension for shell scripts
           files: >
             (?x)^(
                  function/scource/*|
                  scripts/*
             )$
           types: [shell, executable]
         - id: shellcheck
           types: [shell, executable]
  4. In the root directory of the project, create the configuration file for the isort Python library by running this command: touch .isort.cfg
  5. In the new .isort.cfg file, add the following text, and save the file.
    [settings]
    line_length = 88
    multi_line_output = 3
    include_trailing_comma = True
    known_third_party = boto3,crhelper,requests,ruamel
  6. While still in the root directory of the project, create the configuration file for the black Python library by running this command: touch pyproject.toml
  7. In the new pyproject.toml file, add the following text, and save the file. Note that the pyproject.toml file excludes certain directories when the Quick Start doesn’t include Python files.
    [tool.black]
     line-length = 88
     target_version = ['py36', 'py37', 'py38']
     include = '\.py'
     exclude = '''
       /(
           \.eggs         # exclude a few common directories in the
         | \.git          # root of the project
         | templates
         | docs
       )/
     '''
  8. Still in the root directory of the project, create the configuration file for the flake8 Python library by running this command: touch .flake8
  9. In the new .flake8 file, add the following text, and save the file. Note that we added a line to let flake8 know which errors and warning codes to ignore when it does its magic. To see a complete list of codes, see the pycodestyle documentation.
    [flake8]
     max-line-length = 100
     max-complexity = 18
     select = B,C,E,F,W,T4,B9
     ignore = E203, E266, E501, W503, F403, F401

Step 4: Install the pre-commit hooks

  1. Navigate to the root of your forked Quick Start directory, and install the hooks from the .pre-commit-config.yaml file by running the following command: pre-commit install
  2. If you add or remove hooks in the .pre-commit-config.yaml file, apply the changes by running the following command: git add .pre-commit-config.yamlIf you don’t run that command after adding or removing hooks, then when you use your pre-commit hooks, you’ll receive the message “[ERROR] Your pre-commit configuration is unstaged. `git add .pre-commit-config.yaml` to fix this,” as shown in Figure 2.

    error message

    Figure 2. Error message you may receive

Step 5: Test your pre-commit hooks

  1. Run the pre-commit hooks against all the files in the Quick Start project with the following command: pre-commit run --all-filesThis command identifies all errors in the project at one time. If you encounter a similar naming issue in another Quick Start project, determine how shell scripts are called in the code. If you add the .sh file extension to your shell scripts, either update your template files or exclude those files from the pre-commit hook check.
  2. Test a commit for errors as follows.
    (a) In the templates directory of the Quick Start, add Default: 10.0.0.1/24 to the Parameters section in the amazon-eks-master.template.yaml file. Figure 3 gives an example of what your file might look like after you add this text:

    example file content

    Figure 3. Example file content

    (b) Add and commit your text addition to Git by running the following commands:

    git add amazon-eks-master.template.yaml
     $ git commit -m "Updated Parameter RemoteAccessCIDR"

    Your output shows that the tests that passed or were skipped, as shown in Figure 4:

    tests-passed

    Figure 4. Tests passed or skipped

Cleanup

To clean up from the previous exercises, remove the forked repo and your local clone as follows.

  1. Remove your local clone of the repo by running the following command: rm -Rf /path/to/QuickStart
  2. Remove the forked repo from your GitHub account as follows.
    a. Navigate to your forked repo page, and choose Settings.
    b. IMPORTANT: At the bottom of the page, in the Danger Zone section, confirm that you’re in your forked repo, not the primary Quick Start for Amazon EKS Architecture repo. You know that you’re in the forked repo because the path includes your account name:<your-account>/quickstart-amazon-eks
    c. Choose Delete this repository.
    d. Enter the name of your forked repo.
  3. (Optional) Remove the pre-commit hooks by running this command: pip uninstall pre-commit

Conclusion

Walking through the exercise in this post gives you a taste of how pre-commit hooks can save you time and frustration. We hope that you’re now thinking of ways to use pre-commit hooks with your AWS CloudFormation projects and others. As you use the hooks, keep exploring ways that they can benefit your work. If you feel inspired, develop your own pre-commit hook!

To learn more about pre-commit hooks and how you can use them to make your life easier, see the pre-commit documentation.

Please let us know your thoughts in the comments.