1

When we create a parquet file with append option the first partition file of the parquet is missing from the final result. Any one know the reason. we are using Dask 2.30. And this happens only in one environment but in another completely different environment the issue doesnt occurs. Any one experience such an issue. I found a work around instead of creating the folder by dask , i created using a makedir option before it writes and give a few mins of sleep its works fine.

Arun
  • 41
  • 1
  • 4
  • 1
    Please post some sample code for us to work from. Your description makes it seem like you might have version incompatibilities (dask 2.30 is rather old), so it would be helpful if you give all versions too. – mdurant Jan 21 '22 at 18:47
  • sorry for the late response, actually its not a dask issue, its an issue with azure file storage (standard one), since there is a small latency before the first part of the file write to the storage the next part replaces it (this part files write in a loop which executes in milli second gap) – Arun Feb 05 '22 at 17:19
  • Perhaps you should answer your own question in this case, showing how you worked around it. – mdurant Feb 05 '22 at 17:23

0 Answers0