In one of my scripts I call the following code on my dataframe to save the data on disc.
to_csv(input_folder / 'tmp' / section_fname, index=False, encoding='latin1', quoting=csv.QUOTE_NONNUMERIC)
When I opened the created file with notepad++ in the "show all characters mode" it showed a lot of NUL characters (\0) inside one of the rows. In addition to this, some rows of the dataframe are not being written
However, if I scroll this line, there are some data of my dataframe after:
This appears somewhere in the middle of my dataframe, so I decided to call head and then tail to look inside this specific portion of the data where it appears. As I can see, the data is pretty all right: there are some integers and strings as it should be.
I am using pandas 1.1.5
I have looked throught the data to ensure that nothing weird is being written that can result in reading this way. In addition to this, I have googled if someone faced the same issue, but mostly it occurs that people read the data with pandas and get NUL characters
I have spent a lot of time digging into the data and the code and have no explanation of such behavior. Maybe someone can help me?
By the way, everytime I write my dataframe it occurs in a different place removing different amount of rows.
Kind regards, Mike
