2

Do you know how I can fix this problem in PyTorch 1.9?

File "main.py", line 138, in main
    checkpoint = torch.load(args.resume)
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/serialization.py", line 608, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/serialization.py", line 787, in _legacy_load
    result = unpickler.load()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 2: invalid start byte

I have:

$ pip freeze
h5py==3.3.0
joblib==1.0.1
numpy==1.21.2
Pillow==8.3.1
scikit-learn==0.24.2
scipy==1.7.1
sklearn==0.0
threadpoolctl==2.2.0
torch==1.9.0
torchaudio==0.9.0
torchvision==0.10.0
typing-extensions==3.10.0.0
Mona Jalal
  • 34,860
  • 64
  • 239
  • 408
  • 1
    Was the file you're trying to load saved by your code? I've seen this happening when the model is very old (PyTorch 0.4 or older) or was saved in Python 2. Does [this](https://stackoverflow.com/questions/61851244/getting-unicode-decode-error-while-trying-to-load-pre-trained-model-using-torch) help? – Berriel Aug 20 '21 at 21:04
  • 1
    I don't think it's a problem with PyTorch. Are you sure you have non-ascii characters in your path? Maybe in your username? – Corralien Aug 20 '21 at 21:08
  • So the code has been working with previous versions of PyTorch. I switched to current new stable version of PyTorch, 1.9 and now it is not working. I haven't done any modification to code or data and am just running it as is. @Corralien – Mona Jalal Aug 20 '21 at 21:10
  • @Berriel i am trying to run the code mentioned here with no change to code or data though with PyTorch 1.9 https://github.com/mvasil/fashion-compatibility – Mona Jalal Aug 20 '21 at 21:10
  • 1
    @MonaJalal as it is written in their repo: the code was tested using "_Pytorch version 0.1.12_" which is VERY old. Have you tried the solution I linked in my previous comment? [This one.](https://stackoverflow.com/a/67158943/4228275) BTW, you'll need to fix many other things to run it on PyTorch 1.9 – Berriel Aug 20 '21 at 21:13

1 Answers1

3

Can you try something like this:

>>> torch.load('model.pt', encoding='ascii')  # or latin1, or other encoding

By default, we decode byte strings as utf-8. This is to avoid a common error case UnicodeDecodeError: 'ascii' codec can't decode byte 0x... when loading files saved by Python 2 in Python 3. If this default is incorrect, you may use an extra encoding keyword argument to specify how these objects should be loaded, e.g., encoding='latin1' decodes them to strings using latin1 encoding, and encoding='bytes' keeps them as byte arrays which can be decoded later with byte_array.decode(...).

Read this, the 2nd note.

Corralien
  • 109,409
  • 8
  • 28
  • 52