Managing AWS Credentials with Boto3 in Python – Complete guide

This article is a must-read for anyone who interacts with AWS using Python, and particularly the Boto3 library.

Configuring credentials in Boto3 might seem like a straightforward task at first, but, in reality, it is a critical process that demands meticulous handling.

Why? Because the way you set up and manage your AWS credentials can have significant impacts on the security of your AWS resources and the efficiency of your development workflows.

In this guide, we will walk you through four methods of specifying credentials in Boto3, starting from the basic approaches of using environment variables and shared credential files to the more advanced and scalable solutions of AWS Config file and AWS IAM Identity Center.

Prerequisites

Before you can start using boto3 on you AWS Account, you’re required to have done the following prerequisites before you can interact with AWS Services using Boto3 with your credentials:

• Install Python3 and Boto3 on your system
• Install the AWS CLI and configure an AWS profile

1. Install Python3

Python3 installation differs based on your operating system:

For Windows

Download the official Python3 installer from the Python website here.

Run the installer file and follow the prompts, make sure to check the box that says “Add Python to PATH” before clicking Install Now.

For MacOS

MacOS comes with Python 2.7 by default, but you can install Python3 using Homebrew.

If you don’t have Homebrew installed, you can install it by pasting the following command in your terminal:

 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Once Homebrew is installed, you can install Python3 by running:

brew install python3

For Linux

Open your terminal and update the package list using:

sudo apt update

Install Python3 by running:

sudo apt install python3

You can verify your Python installation by running python --version in your terminal. You should see a response with the Python version number.

2. Installing Boto3

Now that Python3 is installed, you can install Boto3. The process is the same for all operating systems:

1. Open your terminal.
2. Install Boto3 using pip, which is a package manager for Python. Run the following command: pip3 install boto3

That’s it! You’ve installed Boto3 on your system.

You can verify the installation by opening your Python interpreter with the command python, and then try to import the boto3 module using import boto3.

If you don’t see any error messages, Boto3 was installed successfully.

3. Install AWS CLI and configure an AWS profile

The AWS CLI is a command line tool that allows you to interact with AWS services in your terminal. Depending on if you’re running Linux, macOS, or Windows the installation goes like this:

# macOS install method:
brew install awscli

# Windows install method:
wget https://awscli.amazonaws.com/AWSCLIV2.msi
msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi

# Linux (Ubuntu) install method:
sudo apt install awscli

In order to access your AWS account with the AWS CLI, you first need to configure an AWS Profile. There are 2 ways of configuring a profile:

• Access and secret key credentials from an IAM user
• AWS Single Sign-on (SSO) user

In this article, I’ll briefly explain how to configure the first method so that you can proceed more quickly to set up the Amazon S3 Bucket in AWS CDK.

If you wish to set up the AWS profile more securely, then I’d suggest you read and apply the steps described in setting up AWS CLI with AWS Single Sign-On (SSO).

In order to configure the AWS CLI with your IAM user’s access and secret key credentials, you need to login to the AWS Console.

Go to IAM > Users, select your IAM user and click on the Security credentials tab to create an access and secret key.

Then configure the AWS profile on the AWS CLI as follows:

➜ aws configure
AWS Access Key ID [None]: <insert_access_key>
AWS Secret Access Key [None]: <insert_secret_key>
Default region name [None]: <insert_aws_region>
Default output format [json]: json

Your was credentials are stored in ~/.aws/credentials and you can validate that your AWS profile is working by running the command:

➜ aws sts get-caller-identity
{
    "UserId": "AIDA5BRFSNF24CDMD7FNY",
    "Account": "012345678901",
    "Arn": "arn:aws:iam::012345678901:user/test-user"
}

How Boto3 chooses which method to use first when multiple credential methods are available

In Boto3, configuration data is essentially bifurcated into two categories: credentials and non-credentials.

Items such as aws_access_key_id, aws_secret_access_key, and aws_session_token fall under the umbrella of credentials. These are the keys that allow you to authenticate your requests when using Boto3.

The non-credential configurations, on the other hand, contain data that isn’t necessarily related to access permissions but are integral to how Boto3 interacts with AWS services.

These include settings like the region in which your resources reside, or the addressing style that Amazon S3 should use.

When Boto3 needs to authenticate a request, it goes on a hunt for the appropriate credentials.

This hunt involves a systematic check of several possible locations where these credentials could be stored. As soon as Boto3 finds valid credentials, it ceases its search.

The order of precedence when Boto3 searches for these credentials is as follows:

1. Passing credentials as parameters in the boto.client() method
2. Passing credentials as parameters when creating a Session object
3. Environment variables
4. Shared credential file (~/.aws/credentials)
5. AWS config file (~/.aws/config)
6. Assume Role provider
7. Boto2 config file (/etc/boto.cfg and ~/.boto)
8. Instance metadata service on an Amazon EC2 instance that has an IAM role configured.

Down below you’ll find 4 methods of specifying your credentials when connecting to AWS Services using Boto3.

Each method explains the pro’s and cons of the implementation.

Method 1: Using AWS credentials file

The AWS credentials file is a text file on your local machine that stores your AWS access keys. By default, it is located in ~/.aws/credentials.

You can create and configure the AWS credentials file manually or use the AWS CLI command aws configure.

To connect to an AWS service, you can use Boto3 like this:

import boto3

s3 = boto3.resource('s3')

Boto3 will automatically look for AWS credentials in your credentials file.

Method 2: Using environment variables

Environment variables are a way to store key-value pairs in the environment of the operating system.

You can set AWS credentials as environment variables like this:

export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key

Once these environment variables are set, you can connect to an AWS service using Boto3 like this:

import boto3

s3 = boto3.resource('s3)

Boto3 will automatically look for AWS credentials in your environment variables.

Method 3: Using IAM Identity Center

In AWS, the IAM Identity Center, which is the successor to AWS Single Sign-On (SSO), has a unique feature that lets you manage your SSO profiles and login sessions.

This feature was introduced in version 1.14.0 of Boto3 and provides support for single sign-on (SSO) credentials.

Using profiles in your shared configuration file (~/.aws/config). These profiles could be one of the following:

# SSO Profile
[profile my-sso-profile]
sso_start_url = https://my-sso-portal.awsapps.com/start
sso_region = eu-west-1
sso_account_id = 123456789011
sso_role_name = Administrator

# IAM Role Profile
[profile my-iam-role]
role_arn=arn:aws:iam::123456789011:role/Administrator
source_profile=my-sso-profile
region=eu-west-1

Now, to create a Boto3 session, you can specify the profile name via the AWS_PROFILE environment variable or use the profile_name argument when creating a Session.

The following Python code demonstrates this:

import boto3

session = boto3.Session(profile_name='my-sso-profile')
s3_client = session.client('s3')

In this code snippet, we first import the Boto3 module. We then create a new session using boto3.Session() and specify the profile name 'my-sso-profile'.

Finally, we use this session to create an S3 client. This S3 client will use the credentials associated with the 'my-sso-profile' profile.

Method 4: Providing credentials directly in code

Disclaimer: Don’t use this method! Let me tell you why it’s risky.

Embedding credentials directly in your Python scripts might expose them to unauthorized access, especially when the scripts are committed to version control systems.

If you do choose to use this method, please make sure not to expose the script to public repositories:

import boto3

s3 = boto3.client(
    's3',
    aws_access_key_id='ACCESS_KEY',
    aws_secret_access_key='SECRET_KEY',
)

Conclusion

When it comes to setting up credentials for Boto3 to connect to AWS services, you have a multitude of options at your disposal. Each method we’ve discussed offers its unique benefits and use-cases.

However, among these methods, using the AWS IAM Identity Center (Method 4) shines as the most robust and scalable solution.

Why is it the best? By using the IAM Identity Center, you can leverage the power of AWS Single Sign-On (SSO), allowing for enhanced security, streamlined credential management, and easy profile switching.

This approach not only facilitates the efficient management of multiple AWS accounts but also strengthens the overall security by narrowing the attack surface that could be exploited due to credential mishandling.

Moreover, the ability to directly specify these profiles when creating a Boto3 session simplifies the task of maintaining different sessions for different AWS environments.