7

When trying to read a file from s3 with joblib.load() I get the error ValueError: embedded null byte when attempting to read files.

The files were created by joblib and can be successfully loaded from local copies (that were made locally before uploading to s3), so the error is presumably in storage and retrieval protocols from S3.

Min code:

####Imports (AWS credentials assumed)
import boto3
from sklearn.externals import joblib


s3 = boto3.resource('s3')
bucket_str = "my-aws-bucket"
bucket_key = "some-pseudo/folder-set/my-filename.joblib"
joblib.loads(s3.Bucket(bucket_str).Object(bucket_key).get()['Body'].read())
Mark_Anderson
  • 1,229
  • 1
  • 12
  • 34

3 Answers3

11

The following code reconstructs a local copy of the file in memory before feeding into joblib.load(), enabling a successful load.

from io import BytesIO
import boto3
from sklearn.externals import joblib

s3 = boto3.resource('s3')
bucket_str = "my-aws-bucket"
bucket_key = "some-pseudo/folder-set/my-filename.joblib"
with BytesIO() as data:
    s3.Bucket(bucket_str).download_fileobj(bucket_key, data)
    data.seek(0)    # move back to the beginning after writing
    df = joblib.load(data)

I assume, but am not certain, that something in how boto3 chunks files for download creates a null byte that breaks joblib, and BytesIO fixes this before letting joblib.load() see the datastream.

PS. In this method the file never touches the local disk, which is helpful under some circumstances (eg. node with big RAM but tiny disk space...)

Mark_Anderson
  • 1,229
  • 1
  • 12
  • 34
1

You can do it like this using the s3fs package.

import s3fs

fs = s3fs.S3FileSystem() # Updated method name
filename = "s3://<bucket_name>/<path_to>/<my_file>.joblib>"
with fs.open(filename, encoding='utf8') as fh:
    data = joblib.load(fh)

I guess everybody has their own preference but I really like s3fs because it makes the code look very familiar to people who haven't worked with s3 before.

Jelmer Wind
  • 342
  • 3
  • 10
  • Awesome! I made some corrections in another answer - feel free to change yours and I'll remove mine. Thanks! – jtlz2 May 15 '23 at 09:09
1

@Jelmer Wind's answer is great but contains some errors - here is an updated version that was a bit long for a comment:

import s3fs

fs = s3fs.S3FileSystem() # Updated method name
filename = "s3://<bucket_name>/<path_to>/<my_file>.joblib>"
with fs.open(filename, encoding='utf8') as fh:
    data = joblib.load(fh)
jtlz2
  • 7,700
  • 9
  • 64
  • 114