2

I have a .json file on Windows which I tried to commit to my git repo. We have the end-of-file-fixer pre-commit hook activated, to make sure there is always a new line at the end of each file. However, each time I commit my file it adds some unknown character to the end of my file:

enter image description here

If I later try to read this file with Python, it complains that there is an unknown character in the json and it can't decode the file.

Traceback (most recent call last):
  File "C:\Users\rruiter\Miniconda3\envs\nest39\lib\site-packages\IPython\core\interactiveshell.py", line 3251, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-cd035d98e6fe>", line 2, in <module>
    data = json.loads(f.read())
  File "C:\Users\rruiter\Miniconda3\envs\nest39\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Users\rruiter\Miniconda3\envs\nest39\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\rruiter\Miniconda3\envs\nest39\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
anthony sottile
  • 61,815
  • 15
  • 148
  • 207
Roald
  • 2,459
  • 16
  • 43
  • 2
    Note that *Git* does not have an "end of file fixer". That's some add-on you installed as a hook, that doesn't come with Git itself. But the UTF-16 file format would explain the problem: Windows is about the only system that uses UTF-16, which is a pretty dumb file format to use (to be fair to Microsoft this wasn't so obvious at the time they started using it, more than 20 years ago). Note also that the error you're getting has nothing to do with the final newline—it's just the fact that the json reader doesn't like UTF-16! – torek Feb 21 '22 at 10:59

1 Answers1

4

Turns out my my .json file was encoded as UTF-16 LE BOM. When I converted it into UTF-8 everything worked again.

Roald
  • 2,459
  • 16
  • 43