Get objects

Load Python objects from an S3 bucket to Python

The following function can be used to get a Python object in an S3 bucket to Python. This works for any kind of Python object stored in S3, like sklearn models, arrays, pandas dataframes etc.

from io import BytesIO
import joblib
import boto3

def get_obj(key, bucket):
    try:
        # memory buffer
        buffer = BytesIO()
        # download object from S3-bucket
        client = boto3.client('s3')
        client.download_fileobj(
            Fileobj=buffer,
            Bucket=bucket,
            Key=key,
        )
        # reset buffer position
        buffer.seek(0)
        # return object
        if buffer is None:
            return None
        return joblib.load(buffer)
    except:
        pass

When calling the function you need to pass

• key - the key ("filepath") of the object you want to get from the S3 bucket

• bucket - the name of the bucket

key = 'key_name'
bucket = 'bucket_name'

# get python object from S3-bucket
obj = get_obj(key=key, bucket=bucket)