How to fix boto3 upload_fileobj TypeError: Strings must be encoded before hashing

Problem:

You are trying to use boto3’ upload_fileobj() to upload a file to S3 storage using code such as

# Create connection to Wasabi / S3
s3 = boto3.resource('s3',
    endpoint_url = 'https://minio.mydomin.com',
    aws_access_key_id = 'ACCESS_KEY',
    aws_secret_access_key = 'SECRET_KEY'
)
# Get bucket object
my_bucket = s3.Bucket('my-bucket')
# Upload string to file
with open("testtext.txt", "r") as f:
    my_bucket.upload_fileobj(f, "test.txt")

But when you try to run it, you see the following stacktrace:

Traceback (most recent call last):
  File "/home/uli/dev/MyProject/put.py", line 13, in <module>
    my_bucket.upload_fileobj(f, "test.txt")
  File "/usr/local/lib/python3.10/dist-packages/boto3/s3/inject.py", line 678, in bucket_upload_fileobj
    return self.meta.client.upload_fileobj(
  File "/usr/local/lib/python3.10/dist-packages/boto3/s3/inject.py", line 636, in upload_fileobj
    return future.result()
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/s3transfer/upload.py", line 758, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/usr/local/lib/python3.10/dist-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/dist-packages/botocore/client.py", line 933, in _make_api_call
    handler, event_response = self.meta.events.emit_until_response(
  File "/usr/local/lib/python3.10/dist-packages/botocore/hooks.py", line 416, in emit_until_response
    return self._emitter.emit_until_response(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/botocore/hooks.py", line 271, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/usr/local/lib/python3.10/dist-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/botocore/utils.py", line 3088, in conditionally_calculate_md5
    md5_digest = calculate_md5(body, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/botocore/utils.py", line 3055, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/usr/local/lib/python3.10/dist-packages/botocore/utils.py", line 3068, in _calculate_md5_from_file
    md5.update(chunk)
TypeError: Strings must be encoded before hashing

Solution

You need to open the file in binary mode  ("rb") . Instead of

with open("testtext.txt", "r") as f:

use

with open("testtext.txt", "rb") as f:

This will fix the issue.