Validating CloudFormation templates in AWS CodePipeline

Validating CloudFormation templates in AWS CodePipeline

Written on August 27, 2020 by Danny Steenman.

Last updated: February 02, 2021.

5 min read

CloudFormation Linter (cfn-lint) is a static code analysis tool that validates CloudFormation YAML and JSON templates against the CloudFormation Resource Specification. Engineers who work with CloudFormation normally run this tool locally on their machines to fix any issues with the templates and correct them immediately.

However... there are use cases where you work in bigger teams and need to manage hundreds of templates. Integrating cfn-lint in a CI/CD pipeline helps to enforce rules over shared CloudFormation templates, which in turn makes standardizing rules and guidelines over bigger teams easier. We are going to be integrating cfn-lint in AWS CodePipeline using a custom AWS CodeBuild project.

Overview

AWS CloudFormation CodePipeline linter diagram

In this example, we have an AWS CodePipeline that contains the source code including the CloudFormation templates in an AWS CodeCommit repository. Next up we have the Validation stage where we have an AWS CodeBuild action that contains the cfn-lint tool. The templates will be scanned and the report will be sent to an encrypted S3 bucket. The last stage contains the actual deployment after the validation has finished.

Pipeline

If we're building the pipeline it looks as follows (snippet):

CodePipeline: Type: AWS::CodePipeline::Pipeline Properties: ArtifactStore: Location: Ref: ArtifactStore Type: S3 Name: !Ref CloudFormationPipelineName RoleArn: !GetAtt CodePipelineRole.Arn RestartExecutionOnUpdate: False Stages: - Name: Source Actions: - Name: GetSource ActionTypeId: Category: Source Owner: AWS Version: 1 Provider: CodeCommit OutputArtifacts: - Name: source Configuration: BranchName: master RepositoryName: cloudformation-codecommit-repo - Name: Validation Actions: - Name: Linter RunOrder: 1 InputArtifacts: - Name: source ActionTypeId: Category: Build Owner: AWS Version: 1 Provider: CodeBuild Configuration: ProjectName: !Ref CodeBuildLinter - Name: Deploy Actions: - Name: Dev RunOrder: 1 InputArtifacts: - Name: source ActionTypeId: Category: Build Owner: AWS Version: 1 Provider: CodeBuild Configuration: ProjectName: !Ref CodeBuildDev ArtifactStore: Type: AWS::S3::Bucket Properties: BucketEncryption: ServerSideEncryptionConfiguration: - ServerSideEncryptionByDefault: SSEAlgorithm: aws:kms
yaml

The CloudFormation template snippet contains the three stages that were mentioned before Source -> Validation -> Deploy including an artifact bucket to save the artifacts from the build.

CodeBuild

The most important part of this pipeline is the validation stage which contains the CodeBuild step:

CodeBuildLinter: Type: AWS::CodeBuild::Project Properties: Artifacts: Type: CODEPIPELINE Description: Linting CloudFormation templates and sending reports to S3 Environment: ComputeType: BUILD_GENERAL1_SMALL EnvironmentVariables: - Name: AWS_ACCOUNT_ID Value: !Sub ${AWS::AccountId} - Name: CFN_LINT_BUCKET Value: !Ref CfnLintBucket - Name: CFN_LINT_KMS_KEY_ARN Value: !Ref CfnLintKmsKeyId - Name: CODE_PIPELINE_NAME Value: !Ref ProvisioningPipelineName Image: aws/codebuild/standard:3.0 Type: LINUX_CONTAINER ServiceRole: !GetAtt CodeBuildRole.Arn Source: Type: CODEPIPELINE BuildSpec: buildspec-linter.yml TimeoutInMinutes: 20
yaml

The cfn-lint tool is added as a CodeBuild action in the validation stage of this pipeline. We add additional environment variables like the AWS Account ID, S3 report bucket, KMS Key and the pipeline name. The next step is to set up the buildspec of the CodeBuild project and add the container with the commands that need to be executed:

version: 0.2 phases: install: runtime-versions: python: 3.8 commands: - apt-get update -y - apt-get install -y uuid-runtime - pip install cfn-lint build: commands: - | REPORT=$(uuidgen)-cfnlint-report.json set +e; cfn-lint -f json > $REPORT aws s3 cp $REPORT s3://$CFN_LINT_BUCKET/$AWS_ACCOUNT_ID/$CODE_PIPELINE_NAME/$(date +'%Y')/$(date +'%m')/$(date +'%d')/ --sse=aws:kms --sse-kms-key-id $CFN_LINT_KMS_KEY_ARN --acl bucket-owner-full-control
bash

First, we install the required packages in the container. Next, we run cfn-lint and make sure the command won't exit (set +e) when it reports an error. In this situation, we don't want to stop exit CodeBuild if we discover an error because we use cfn-lint only for reporting.

In this setup, I've included a configuration file which specifies what templates need to be analyzed.

After the analysis, it will generate a report in JSON format which will be uploaded to an encrypted S3 bucket. As you can see the S3 CLI command is being used to transfer the report to the S3 bucket. We've included the variables which we passed from the AWS::CodeBuild::Project resource to dynamically create the object and enable KMS Encryption. After the report is uploaded to S3 it is ready for consumption for other applications.

KMS & S3 Bucket

If you enable AWS KMS, make sure to include the following policies for the CodeBuild role:

- Action: - s3:ListBucket - s3:ListBucketVersions - s3:ListObjects - s3:PutObject - s3:PutObjectAcl Effect: Allow Resource: - !GetAtt CfnLintBucket.Arn - !Sub ${CfnLintBucket.Arn}/* Sid: Allow S3 put CfnLintBucket - Action: - kms:Encrypt - kms:ReEncrypt* - kms:GenerateDataKey* - kms:DescribeKey - kms:ListKeys Effect: Allow Resource: !GetAtt KMSKey.Arn Sid: Allow use of CMK in CfnLintBucket
yaml

The CodeBuild role is allowed to put the report in the destination bucket (CfnLintBucket). We also allow the role to encrypt the object using the KMSKey we created for the S3 Bucket. Then we need to explicitly allow the CodeBuildRole on the KMS Key policy, we can do that as follows:

- Sid: Allow encrypt use of the key by the CodeBuildRole Effect: Allow Principal: AWS: - !GetAtt CodeBuildRole.Arn Action: - kms:Encrypt - kms:ReEncrypt* - kms:GenerateDataKey* - kms:DescribeKey - kms:ListKeys Resource: '*'
yaml

To Summarize

We've set up a CodePipeline which adds cfn-lint as a validation stage to analyze your CloudFormation templates and send the reports to an encrypted S3 Bucket. This allows auditors for example to check how well the templates are built according to best practices.

Share on Twitter

Subscribe to the newsletter

If you're interested in AWS Cloud, Infrastructure as Code, DevOps, and getting certified in AWS then subscribe to my newsletter to get exclusive tips and tricks on becoming a successful Cloud Engineer.

- subscribers