3

I have some py scripts to read data from db and write it to a feather file (with pandas to_feather() method). But somehow the written file cannot be read and it throws error as below:

Traceback (most recent call last):
  File "/opt/***/***pipeline.py", line 303, in <module>
    __main__()
  File "/opt/***/***pipeline.py", line 124, in __main__
    imt_preprocess.preprocess_data(config, thread_id=None, filtered_durations=durations, module=module, output_directory=output_directory, valid_from=valid_from, valid_until=valid_until)
  File "/opt/***/***modules/imt_preprocess.py", line 27, in preprocess_data
    df = commons.read_data(source_path)
  File "/opt/***/***commons.py", line 62, in read_data
    df = pd.read_feather(path)
  File "/home/***/.local/lib/python3.7/site-packages/pandas/io/feather_format.py", line 131, in read_feather
    handles.handle, columns=columns, use_threads=bool(use_threads)
  File "/home/***/.local/lib/python3.7/site-packages/pyarrow/feather.py", line 220, in read_feather
    return (read_table(source, columns=columns, memory_map=memory_map)
  File "/home/***/.local/lib/python3.7/site-packages/pyarrow/feather.py", line 241, in read_table
    reader = _feather.FeatherReader(source, use_memory_map=memory_map)
  File "pyarrow/_feather.pyx", line 75, in pyarrow._feather.FeatherReader.__cinit__
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 114, in pyarrow.lib.check_status
OSError: Verification of flatbuffer-encoded Footer failed.

Any idea what is the reason of this?

Atacan
  • 55
  • 8
  • Were you able to solve this in the end? – Sephos Aug 16 '23 at 08:07
  • 1
    No, we couldn't find the root cause. i think it's a kind of bug. it isn't reproducible. same dataset is read without error after rewriting to feather file. interesting part of the problem is that there is no error or warning when you write to feather. But when you try to read this file you get the error above. we made a workaround solution in py script. after writing file we read same file. if it can be read successfully, go ahead. otherwise re-write to feather until it can be read without error. – Atacan Aug 18 '23 at 08:12

0 Answers0