ValueError: Comments are not supported by the python backend

Question

The ijson module has a documented option allow_comments=True, but when I include it, an error message is produced:

Below is a transcript using the file test.py:

import ijson
for o in ijson.items(open(0), 'item'):
     print(o)

Please note that I have no problem with a similar documented option, multiple_values=True.

Transcript

$ python3 --version
Python 3.10.9
$ python3 test.py <<< [1,2]
1
2
# Now change the call to: ijson.items(open(0), 'item', allow_comments=True)
$ python3 test.py <<< [1,2]
Traceback (most recent call last):
  File "/Users/user/test.py", line 5, in <module>
    for o in ijson.items(open(0), 'item', allow_comments=True):
  File "/usr/local/lib/python3.10/site-packages/ijson/utils.py", line 51, in coros2gen
    f = chain(events, *coro_pipeline)
  File "/usr/local/lib/python3.10/site-packages/ijson/utils.py", line 29, in chain
    f = coro_func(f, *coro_args, **coro_kwargs)
  File "/usr/local/lib/python3.10/site-packages/ijson/backends/python.py", line 284, in basic_parse_basecoro
    raise ValueError("Comments are not supported by the python backend")
ValueError: Comments are not supported by the python backend
$

score 1 · Accepted Answer · answered Jan 09 '23 at 18:55

1

Take a look at the Backends section of the documentation, which says:

Ijson provides several implementations of the actual parsing in the form of backends located in ijson/backends:

yajl2_c: a C extension using YAJL 2.x. This is the fastest, but might require a compiler and the YAJL development files to be present when installing this package. Binary wheel distributions exist for major platforms/architectures to spare users from having to compile the package.

yajl2_cffi: wrapper around YAJL 2.x using CFFI.

yajl2: wrapper around YAJL 2.x using ctypes, for when you can’t use CFFI for some reason.

yajl: deprecated YAJL 1.x + ctypes wrapper, for even older systems.

python: pure Python parser, good to use with PyPy

And later on in the FAQ it says:

Q: Are there any differences between the backends?

... The python backend doesn’t support allow_comments=True It also internally works with str objects, not bytes, but this is an internal detail that users shouldn’t need to worry about, and might change in the future.

If you want support for allow_comments=True, you need to be using one of the yajl based backends. According to the docs:

Importing the top level library as import ijson uses the first available backend in the same order of the list above, and its name is recorded under ijson.backend. If the IJSON_BACKEND environment variable is set its value takes precedence and is used to select the default backend.

You'll need the necessary libraries, etc, installed on your system in order for this to work.

answered Jan 09 '23 at 18:55

larsks

277,717
41
399
399

Thanks. My Google searches missed the FAQ. Anyway, in accordance with the FAQ I tried adding: import ijson.backends.yajl2_cffi as ijson This of course requires YAJL, but `pip3 install YAJL` results in numerous failures, ending with: note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure × Encountered error while trying to install package. ╰─> YAJL note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure. – peak Jan 09 '23 at 19:08
Do you have the yajl C libraries installed on your system? That may be as simple as `yum install yajl` or `apt install libyajl2`. You shouldn't need to explicitly `pip install yajl`, because the Python yajl module isn't required by ijson. You also shouldn't need to explicitly select a backend -- as the docs mention, backends are selected in the order they're listed, so if you have the appropriate dependencies installed you should get a yajl-based parser by default. – larsks Jan 09 '23 at 19:43
`brew install yajl` did it! Thank you!!! Do you happen to know how a Python program can determine whether `yajl` is available dynamically? Or is there some other way to determine if the `allow_comments=True` can be used? – peak Jan 09 '23 at 20:56
You can check the value of `ijson.backend` to see what backend has been selected, and make your code conditional on that. Or just try to parse some test data with `ijson` and respond to the ValueError. – larsks Jan 09 '23 at 21:26
Yes! For reference, an appropriate test would be to test for both "yajl2_cffi" and "yajl2". Tx. – peak Jan 10 '23 at 03:44

ValueError: Comments are not supported by the python backend

Transcript

1 Answers1