How to write a file or data to S3 using Python Boto3

Whether you’re managing large datasets or handling user-generated content, writing files to S3 is a common task that developers encounter.

In this guide, we’ll explore how to write files to S3 using Python’s Boto3 library.

We’ll cover three scenarios: uploading a file directly, writing a string, and writing the contents of a JSON object.

Prerequisites

Before you can start, you’re required to have done the following prerequisites before you can run Python S3 Boto3 calls on your AWS account.

• Install the AWS CLI and configure an AWS profile
• Setting up the Python Environment
• Create an S3 bucket if that doesn’t exist yet

If you’ve already done this, you can proceed to the next section of this article.

1. Install AWS CLI and configure an AWS profile

The AWS CLI is a command line tool that allows you to interact with AWS services in your terminal.

Depending on if you’re running Linux, macOS, or Windows the installation goes as follows:

# macOS install method:
brew install awscli

# Windows install method:
wget https://awscli.amazonaws.com/AWSCLIV2.msi
msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi

# Linux (Ubuntu) install method:
sudo apt install awscli

In order to access your AWS account with the AWS CLI, you first need to configure an AWS Profile. There are 2 ways of configuring a profile:

• Access and secret key credentials from an IAM user
• AWS Single Sign-on (SSO) user

In this article, I’ll briefly explain how to configure the first method so that you can proceed with running the python script on your AWS account.

If you wish to set up the AWS profile more securely, then I’d suggest you read and apply the steps described in setting up AWS CLI with AWS Single Sign-On (SSO).

In order to configure the AWS CLI with your IAM user’s access and secret key credentials, you need to login to the AWS Console.

Go to IAM > Users, select your IAM user, and click on the Security credentials tab to create an access and secret key.

Then configure the AWS profile on the AWS CLI as follows:

➜ aws configure
AWS Access Key ID [None]: <insert_access_key>
AWS Secret Access Key [None]: <insert_secret_key>
Default region name [None]: <insert_aws_region>
Default output format [json]: json

Your was credentials are stored in ~/.aws/credentials and you can validate that your AWS profile is working by running the command:

➜ aws sts get-caller-identity
{
    "UserId": "AIDA5BRFSNF24CDMD7FNY",
    "Account": "012345678901",
    "Arn": "arn:aws:iam::012345678901:user/test-user"
}

2. Setting up the Python Environment

To be able to run the Python boto3 script, you will need to have Python installed on your machine.

Depending on if you’re running Linux, macOS, or Windows the installation goes like this:

# macOS install method:
brew install python

# Windows install method:
wget https://www.python.org/ftp/python/3.11.2/python-3.11.2-amd64.exe
msiexec.exe /i https://www.python.org/ftp/python/3.11.2/python-3.11.2-amd64.exe

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py

# Linux (Ubuntu) install method:
sudo apt install python3 python3-pip

Once you have installed Python, you will need to install the Boto3 library.

You can install Boto3 using pip, the Python package manager, by running the following command in your terminal:

pip install boto3

3. Create an S3 Bucket with the AWS CLI if It Doesn’t Exist Yet

Before writing files to S3, you’ll need to ensure that the target bucket exists. If not, you can create it using the AWS CLI.

Here’s how to check if a bucket exists and create it if necessary:

Run the following command to see the available buckets in your AWS account.

➜ aws s3 ls

2023-05-11 14:52:11 cdk-hnb659fds-assets-eu-west-1

It shows a lists of buckets that are available on your AWS account.

If the bucket doesn’t exist, you can create it with the following command:

➜ aws s3 mb s3://hello-towardsthecloud-bucket-eu-west-1 --region eu-west-1

make_bucket: hello-towardsthecloud-bucket-eu-west-1

Replace 'hello-towardsthecloud-bucket-eu-west-1' with your desired bucket name and 'eu-west-1' with the appropriate AWS region, such as 'us-east-1'.

To ensure that the bucket was created successfully, you can list all your buckets again with: aws s3 ls to see your newly created S3 bucket.

Note: Make sure you’re logged in to the correct AWS CLI profile and have the necessary permissions to create and manage S3 buckets.

By following these steps, you can ensure that the necessary S3 bucket is available for your file-writing operations.

Now you’re ready to proceed with uploading files or writing data to your S3 bucket using Python’s Boto3 library.

How to write objects to an S3 Bucket using Python Boto3

There are multiple ways to write data to an S3 object in an S3 bucket. In this section we’ll go over 3 popular methods to get your data in S3.

• Upload a file directly to S3
• Write a string to a new object in S3
• Write a JSON to a new object in S3

1. Upload a File directly to an S3 Bucket

Uploading a file directly to S3 is a straightforward task with Boto3.

Here’s how you can do it:

import boto3

s3 = boto3.client("s3")

s3.upload_file("local_file.txt", "my-bucket", "object_name.txt")

First, import the Boto3 library using import boto3.

Then create an S3 client using your AWS credentials: s3 = boto3.client('s3')

At last use the upload_file method to upload a file to the specified bucket: s3.upload_file('local_file.txt', 'my-bucket', 'object_name.txt')

Note: Replace 'local_file.txt' with your local file path, and 'my-bucket' with your bucket name.

2. Write String to a New Object in S3

An alternative to uploading files directly is to write data in the form of a string directly to an S3 object.

Use the following example to create the data and use the put method in the s3.Object to place a string in a new object.

import boto3

s3 = boto3.resource("s3")

s3.Object("my-bucket", "object_name.txt").put(
    Body="Hello, World!"
)

The Body contains the actual content of the object and the object_name.txt is where you wish the save the content on the S3 bucket.

3. Write the Contents of a JSON to a New Object in S3

Writing JSON data to S3 can be useful for configuration files, data interchange, and more. Here’s an example of how to do it:

import boto3
import json

s3 = boto3.client("s3")


data = {"key": "value"}
json_str = json.dumps(data)

s3.put_object(
    Bucket="my-bucket",
    Key="object_name.json",
    Body=json_str,
)

Note: Replace 'my-bucket' with your bucket name and adjust the JSON data as needed.

Conclusion

Writing files to Amazon S3 using Python’s Boto3 library is a versatile and essential skill for AWS developers.

In this guide, we’ve explored different methods on how to upload a file directly, write a string to a new object, and write JSON data to S3.

Whether you’re working with large datasets or simply storing configuration files, these methods provide a clear and concise way to interact with S3.