Validating CloudFormation templates in AWS CodePipeline

CloudFormation Linter (cfn-lint) is a static code analysis tool that validates CloudFormation YAML and JSON templates against the CloudFormation Resource Specification.

Engineers who work with CloudFormation normally run this tool locally on their machines to fix any issues with the templates and correct them immediately.

However… there are use cases where you work in bigger teams and need to manage hundreds of templates.

Integrating cfn-lint in a CI/CD pipeline helps to enforce rules over shared CloudFormation templates, which in turn makes standardizing rules and guidelines over bigger teams easier.

We are going to be integrating cfn-lint in AWS CodePipeline using a custom AWS CodeBuild project.

AWS CodePipeline Overview

In this example, we have an AWS CodePipeline that contains the source code including the CloudFormation templates in an AWS CodeCommit repository.

Next up we have the Validation stage where we have an AWS CodeBuild action that contains the cfn-lint tool. The templates will be scanned and the report will be sent to an encrypted S3 bucket.

The last stage contains the actual deployment after the validation has finished.

Creating AWS CodePipeline in CloudFormation

If we’re building the pipeline it looks as follows (snippet):

CodePipeline:
  Type: AWS::CodePipeline::Pipeline
  Properties:
    ArtifactStore:
      Location:
        Ref: ArtifactStore
      Type: S3
    Name: !Ref CloudFormationPipelineName
    RoleArn: !GetAtt CodePipelineRole.Arn
    RestartExecutionOnUpdate: False
    Stages:
      - Name: Source
        Actions:
          - Name: GetSource
            ActionTypeId:
              Category: Source
              Owner: AWS
              Version: 1
              Provider: CodeCommit
            OutputArtifacts:
              - Name: source
            Configuration:
              BranchName: master
              RepositoryName: cloudformation-codecommit-repo
      - Name: Validation
        Actions:
          - Name: Linter
            RunOrder: 1
            InputArtifacts:
              - Name: source
            ActionTypeId:
              Category: Build
              Owner: AWS
              Version: 1
              Provider: CodeBuild
            Configuration:
              ProjectName: !Ref CodeBuildLinter
      - Name: Deploy
        Actions:
          - Name: Dev
            RunOrder: 1
            InputArtifacts:
              - Name: source
            ActionTypeId:
              Category: Build
              Owner: AWS
              Version: 1
              Provider: CodeBuild
            Configuration:
              ProjectName: !Ref CodeBuildDev
ArtifactStore:
  Type: AWS::S3::Bucket
  Properties:
    BucketEncryption:
      ServerSideEncryptionConfiguration:
        - ServerSideEncryptionByDefault:
            SSEAlgorithm: aws:kms

The CloudFormation template snippet contains the three stages that were mentioned before Source -> Validation -> Deploy including an artifact bucket to save the artifacts from the build.

Creating AWS CodeBuild in CloudFormation

The most important part of this pipeline is the validation stage which contains the CodeBuild step:

CodeBuildLinter:
  Type: AWS::CodeBuild::Project
  Properties:
    Artifacts:
      Type: CODEPIPELINE
    Description: Linting CloudFormation templates and sending reports to S3
    Environment:
      ComputeType: BUILD_GENERAL1_SMALL
      EnvironmentVariables:
        - Name: AWS_ACCOUNT_ID
          Value: !Sub ${AWS::AccountId}
        - Name: CFN_LINT_BUCKET
          Value: !Ref CfnLintBucket
        - Name: CFN_LINT_KMS_KEY_ARN
          Value: !Ref CfnLintKmsKeyId
        - Name: CODE_PIPELINE_NAME
          Value: !Ref ProvisioningPipelineName
      Image: aws/codebuild/standard:3.0
      Type: LINUX_CONTAINER
    ServiceRole: !GetAtt CodeBuildRole.Arn
    Source:
      Type: CODEPIPELINE
      BuildSpec: buildspec-linter.yml
    TimeoutInMinutes: 20

The cfn-lint tool is added as a CodeBuild action in the validation stage of this pipeline. We add additional environment variables like the AWS Account ID, S3 report bucket, KMS Key, and the pipeline name.

The next step is to set up the buildspec of the CodeBuild project and add the container with the commands that need to be executed:

version: 0.2
phases:
  install:
    runtime-versions:
      python: 3.8
    commands:
      - apt-get update -y
      - apt-get install -y uuid-runtime
      - pip install cfn-lint
  build:
    commands:
      - |
        REPORT=$(uuidgen)-cfnlint-report.json
        set +e; cfn-lint -f json > $REPORT
        aws s3 cp $REPORT s3://$CFN_LINT_BUCKET/$AWS_ACCOUNT_ID/$CODE_PIPELINE_NAME/$(date +'%Y')/$(date +'%m')/$(date +'%d')/ --sse=aws:kms --sse-kms-key-id $CFN_LINT_KMS_KEY_ARN --acl bucket-owner-full-control

First, we install the required packages in the container. Next, we run cfn-lint and make sure the command won’t exit (set +e) when it reports an error. In this situation, we don’t want to stop exit CodeBuild if we discover an error because we use cfn-lint only for reporting.

In this setup, I’ve included a configuration file that specifies what templates need to be analyzed.

After the analysis, it will generate a report in JSON format which will be uploaded to an encrypted S3 bucket. As you can see the S3 CLI command is being used to transfer the report to the S3 bucket.

We’ve included the variables which we passed from the AWS::CodeBuild::Project resource to dynamically create the object and enable KMS Encryption.

After the report is uploaded to S3 it is ready for consumption for other applications.

Enable AWS KMS & S3 Bucket

If you enable AWS KMS, make sure to include the following policies for the CodeBuild role:

- Action:
    - s3:ListBucket
    - s3:ListBucketVersions
    - s3:ListObjects
    - s3:PutObject
    - s3:PutObjectAcl
  Effect: Allow
  Resource:
    - !GetAtt CfnLintBucket.Arn
    - !Sub ${CfnLintBucket.Arn}/*
  Sid: Allow S3 put CfnLintBucket
- Action:
    - kms:Encrypt
    - kms:ReEncrypt*
    - kms:GenerateDataKey*
    - kms:DescribeKey
    - kms:ListKeys
  Effect: Allow
  Resource: !GetAtt KMSKey.Arn
  Sid: Allow use of CMK in CfnLintBucket

The CodeBuild role is allowed to put the report in the destination bucket (CfnLintBucket). We also allow the role to encrypt the object using the KMSKey we created for the S3 Bucket.

Then we need to explicitly allow the CodeBuildRole on the KMS Key policy, we can do that as follows:

- Sid: Allow encrypt use of the key by the CodeBuildRole
  Effect: Allow
  Principal:
    AWS:
      - !GetAtt CodeBuildRole.Arn
  Action:
    - kms:Encrypt
    - kms:ReEncrypt*
    - kms:GenerateDataKey*
    - kms:DescribeKey
    - kms:ListKeys
  Resource: '*'

To Summarize

We’ve set up a CodePipeline which adds cfn-lint as a validation, stage to analyze your CloudFormation templates and send the reports to an encrypted S3 Bucket.

This allows auditors for example to check how well the templates are built according to best practices.