How can I use "\s+" as a seperator in polars?

Question

So I have a large file of data that's can be up to 11 columns wide, it looks something like this.

1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1

When I read in using pandas i used the code: pd.read_csv(file_dir, skiprows = 1, sep = '\s+'). When pandas reads it, it creates a dataframe that's 3x8(Which is correct). Using Polars: pl.read_csv(file_dir, skip_rows=1, orient='col') When polars reads it, it creates a data frame that's 3x1. I think its due to the separator, but I'm not sure. I tried using "\s+" in polars but it doesn't like it because its bigger than one byte. The delimiter in these files in 7 white spaces.

can you check how the data is stored? with an index or without index? — hilo, Jul 19 '22 at 16:52
@hilo Its not stored using an index, its essentially just a large file with a bunch of numbers formatted like above — 8mrsteel, Jul 19 '22 at 16:57
Sorry , i should have mentioned this, but the first row has the file name in it. I dont want it in the dataframe which is why i skip it — 8mrsteel, Jul 19 '22 at 17:05

score 0 · Answer 1 · answered Jul 19 '22 at 20:26

From the documentation for read_csv and scan_csv, the delimiter must be a single-byte character. As such, you'll need to preprocess your file to convert the 7-whitespace delimiter to a single-byte delimiter before reading it with Polars.

The single-byte limitation is done for performance reasons. For some further background, you can look at this discussion on GitHub. The discussion includes some suggestions for picking single-byte delimiters that are not likely to be found in your input file.

How can I use "\s+" as a seperator in polars?

1 Answers1