Check if a key exists in an S3 Bucket using Boto3 Python

To check if an object exists in a bucket using Boto3, Call the head_object method on the S3 client, passing in the bucket and key. If the key exists, this method will return metadata about the object. If not, it will raise an exception.

Note: If you’re using the AWS CLI to run the Python scripts that are shared in the blog post, then make sure that you can run Python S3 Boto3 calls on your AWS account. Otherwise, you can’t execute any of the code that is shared in this article on your AWS account.

In this guide, you’ll learn how to check if a key exists in a bucket without looping through the whole S3 bucket contents using the Boto3 client. Next to that you’ll also learn an alternative method that let’s you check if multiple objects exist in s3.

How to check if a key exists in an Amazon S3 Bucket using Boto3 Python

To check if a key exists in an S3 bucket with Boto3 Python, you’ll need to follow these 4 steps:

1. Import Boto3: First, import the Boto3 library in your Python script.
2. Create an S3 Client: Utilize Boto3 to create an S3 client that will allow you to interact with the S3 service.
3. Specify the Bucket and Key: Define the bucket name and the key you want to check.
4. Use the head_object Method: Call the head_object method on the S3 client, passing in the bucket and key. If the key exists, this method will return metadata about the object. If not, it will raise an exception.

Here’s a code snippet that puts the above steps into action:

import boto3
import botocore


def key_exists(bucket, key):
    s3 = boto3.client("s3")
    try:
        s3.head_object(Bucket=bucket, Key=key)
        print(f"Key: '{key}' found!")
    except botocore.exceptions.ClientError as e:
        if e.response["Error"]["Code"] == "404":
            print(f"Key: '{key}' does not exist!")
        else:
            print("Something else went wrong")
            raise


bucket = "my-bucket"
key = "path/to/my-file.txt"

key_exists(bucket, key)

To test the code, simply run the script, ensuring that you have the necessary AWS credentials configured. You can modify the bucket and key variables to test different scenarios.

I created an example S3 bucket that contains the following dummy files:

The output looks like follows when I search for a key called ‘hellos3.txt’ in my S3 bucket:

➜ python s3/search_key_in_bucket.py
Key: 'hellos3.txt' found!

You can download this script and find more valuable tools in this GitHub repository.

Best Practices and Considerations

• Error Handling: Make sure to handle exceptions properly, as the head_object method will raise an exception if the key does not exist.
• Permissions: Ensure that the IAM role or user running the script has the necessary permissions to perform the head_object operation on the specified bucket and key.
• Performance: If you need to check multiple keys, consider using other methods like list_objects_v2 to retrieve multiple keys at once. To see an example of how to fetch multiple keys, read the next section.

How to check if multiple keys exists in an Amazon S3 Bucket using Boto3 client

The following approach can be more efficient if you need to check multiple keys at once instead one a single key or object.

import boto3

def check_keys_exist(bucket, keys_to_check):
    s3 = boto3.client('s3')
    response = s3.list_objects_v2(Bucket=bucket)
    
    if 'Contents' in response:
        existing_keys = {item['Key'] for item in response['Contents']}
        return {key: key in existing_keys for key in keys_to_check}
    else:
        return {key: False for key in keys_to_check}

bucket = 'my-bucket'
keys_to_check = ['path/to/file1.txt', 'path/to/file2.txt', 'path/to/file3.txt']

result = check_keys_exist(bucket, keys_to_check)

for key, exists in result.items():
    print(f'Key {key} exists: {exists}')

This code defines a function check_keys_exist that takes a bucket name and a list of keys to check. It uses the list_objects_v2 method to retrieve all the keys in the specified bucket and then checks if the keys in keys_to_check exist within that list.

The result is a dictionary that maps each key to a boolean value indicating whether or not the key exists in the specified bucket.

This approach is more efficient than calling head_object for each key individually, especially when dealing with a large number of keys, as it reduces the number of API calls needed.

To show the result of running the script, I try to look for 2 existing files in my bucket and one non existing:

➜ python s3/search_multiple_keys_bucket.py

Key hellos3.txt exists: True
Key object_name.json exists: True
Key dummy.txt exists: False

It will return either a true or false when it found the specific object and prints the result in the terminal.

You can download this script and find more valuable tools in this GitHub repository.

Conclusion

Checking if a key or multiple keys or objects exists in an S3 bucket is a common task that can be easily achieved using Boto3 in Python.

This guide has provided a step-by-step approach, complete with code examples and best practices.

Whether you’re managing large datasets on S3 or simply need to understand what’s in your bucket. These python scripts can be a valuable tool in your arsenal.