Automating Terraform modules

The benefits of a multi-repo arrangement are:

• Modules can be individually versioned
• Modules can be easily modified without affecting existing infrastructure
• Organization of modules will be easier
• Development of modules can be done without pull requests on the main DevOps repository

I decided to make a template repository to facilitate this process and perform the following automated actions:

• Lint/Format the module using terraform fmt
• Validate the module syntax using terraform validate
• Auto-generate documentation for the module
• Perform a GitHub release for each meaningful merge to the default branch

We’re using CircleCI for these repositories, so this will be specific to that. However, you should be able to adapt these CI actions to any CI platform. In CircleCI, we configure our workflows and jobs in a .circleci/config.yml file. The specifics of this file are documented here. We first configure our executors, the Docker containers that will run the various parts of our workflow. I’ve chosen the CircleCI convenience images for python, node, and golang for the various tasks I want to execute.

version: 2.1

executors:
  docker-terraform:
    docker:
      - image: circleci/python:latest
  node:
    docker:
      - image: circleci/node:latest
  doc:
    docker:
      - image: circleci/golang:latest

Next, we configure our job definitions. First, the job that will test our module in CI for proper format and syntax. If there are errors in Terraform syntax or format, the pull request author will be notified, and the pull request will not be able to be merged into the codebase. For this process, we need to ensure that we’re using the proper version of Terraform, so we use tfenv to control the version. This requires us to define the version of Terraform we’re writing our module for in the main.tf. We also define our provider and default region.

terraform {
 required_version = "0.12.29"
}

provider "aws" {
 version = "~>2.70.0"
 region  = "us-east-1"
}

Testing

The test job definition in CircleCI checks out the module, and then installs tfenv, which we’ll use to install the minimum required version of Terraform as defined in our module, and then we can use the built-in commands to format and validate the module.

jobs:
  test:
    executor: docker-terraform
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true
      - run:
          name: Install tfenv
          command: |
            git clone https://github.com/tfutils/tfenv.git /tmp/.tfenv
            chmod -R +rx /tmp/.tfenv/bin            
      - run:
          name: Install terraform
          command: /tmp/.tfenv/bin/tfenv install min-required
      - run:
          name: Set terraform version
          command: /tmp/.tfenv/bin/tfenv use min-required
      - run:
          name: Run terraform fmt
          command: /tmp/.tfenv/bin/terraform fmt
      - run:
          name: Run terraform init
          command: /tmp/.tfenv/bin/terraform init
      - run:
          name: Run Terraform validate
          command: /tmp/.tfenv/bin/terraform validate

If any of these steps fails, the test job will fail and GitHub and CircleCI will notify us of the failure.

Releasing

In order to take as much manual work out of the module development as we can, we want to automate the creation of GitHub releases. Rather than a naive approach, where we increment the module’s version number by .1 on every merge to the default branch, we want to take a more nuanced approach and use Semantic Release.

Semantic Release will analyze the commit messages that we’re merging to master. Using the Angular Commit Message Guidelines, it will determine if the release should be a bugfix, minor, major, or breaking release, write a release note, and increment the version number of the module. This means that the developer needs to remember to include at least one commit message that follows the format, but also means that merges to the default branch will not always trigger a new release, but only when a new feature, bugfix, or breaking change is added. If developers have trouble remembering to use the proper commit message format, you can add a git commit hook to remind them to do so, but hopefully this won’t be necessary!

To use Semantic Release, the simplest way is to use NodeJS’s npx tool. Adding this step to the pipeline is simple. In .circleci/config.yml we add:

release:
  executor: node
  steps:
    - checkout
    - run: npx semantic-release

And we also need to add a configuration file for semantic-release, .releaserc:

branch: main
branches: ["main"]
plugins:
  [
    "@semantic-release/commit-analyzer",
    "@semantic-release/release-notes-generator",
    "@semantic-release/github",
  ]

This instructs Semantic Release to analyze the commits, generate release notes, and make a new release on GitHub. Semantic Release has other plugins that run by default but we need to make sure they don’t run.

In addition, you’ll need to provide an environment variable for your project in CircleCI called GH_TOKEN or GITHUB_TOKEN that contains a token with write privileges to your repository.

Documentation

Writing documentation for Terraform modules can be difficult, and many modules don’t have documentation for that reason. We want to generate documentation for our Terraform module automatically, so that we can be assured that the model is easy to use for future teams. Terraform-Docs is the only tool I am aware of that performs this task. Here’s how you add the Terraform-Docs configuration to .circleci/config.yml:

documentation:
  executor: doc
  steps:
    - checkout
    - setup_remote_docker:
        docker_layer_caching: true
    - run:
        name: Install terraform-docs
        command: GO111MODULE="on" go get github.com/terraform-docs/terraform-docs@v0.10.0-rc.1
    - run:
        name: Configure github
        command: |
          git config user.email "your_automation_user@your_company.com"
          git config user.name "automation_user"          
    - run:
        name: Run terraform docs
        command: terraform-docs -c .terraform-docs.yml . > README_MODULE.md
    - run:
        name: Check in documentation
        command: |
          git add README_MODULE.md
          git commit -m "[skip ci] Add auto-gen documentation."
          git push https://automation_user:${GH_TOKEN}@github.com/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}.git          

And we place a configuration file for terraform-docs; .terraform-docs.yml:

formatter: markdown table
header-from: main.tf
settings:
  indent: 4

It’s important to note that you should skip the CI run for this commit/push, so you don’t create a loop. You must also use the same GH_TOKEN or GITHUB_TOKEN environment variable you used for the release authentication here.

The pieces are almost complete! Now, we just need a workflow definition for CircleCI, and we can start automatically testing and releasing our Terraform modules.

workflows:
  version: 2
  test-ci:
    jobs:
      - test:
          filters:
            branches:
              ignore: main
  release:
    jobs:
      - test:
          filters:
            branches:
              only: main
      - release:
          requires:
            - test
          filters:
            branches:
              only: main
      - documentation:
          filters:
            branches:
              only: main

Here, we’ve configured CircleCI to run tests on every push of a branch not named main, our default branch, and then tests and documentation on every merge to the main branch, and on a successful test run, to run a release. Even though we should not be merging something that did not pass tests in a developer’s branch, it’s still possible for a branch with an error to get merged to the default branch, so we want to ensure we don’t release in the event that the tests don’t pass.

That’s it! Now each time code is committed to the repository, you can enjoy using a syntactically correct, properly formatted module and auto-generated documentation. Releases will be automated, and not sentimental or romantic. Your team can spend the time they would have used writing this documentation, making manual changes to code formats (indenting, outdenting, etc), and authoring releases on other tasks.