Upload objects

Upload Python objects from Python to an S3 bucket

The following function can be used to upload a Python object from Python to an S3 bucket. This works for any kind of Python object, like sklearn models, arrays, pandas dataframes etc.

from io import BytesIO
import joblib
import boto3

def upload_obj(key, obj, bucket):
    # dump object into a memory buffer
    buffer = BytesIO()
    joblib.dump(obj, buffer)
    # reset buffer position before upload
    buffer.seek(0)
    # upload object to S3-bucket
    client = boto3.client('s3')
    client.upload_fileobj(
        Fileobj=buffer,
        Bucket=bucket,
        Key=key
    )

When calling the function you need to pass

• key - the key ("filepath") you want the object to have in the S3 bucket

• obj - the Python object you want to upload to the S3 bucket

• bucket - the name of the bucket

key = 'key_name'
bucket = 'bucket_name'
# example: Python array
obj = [1, 2, 3]

upload_obj(key=key, obj=obj, bucket=bucket)