-1

I've received a csv file with some issues, I've noticed a few issues when attempting to load into BigQuery. I'm using duckdb to quickly sanitise the data, and I'm noticing a bunch of newline characters in my data.

Is there a quick way to remove newlines from duckdb data before I write back out to csv?

Mikhail Berlyant
  • 165,386
  • 8
  • 154
  • 230
Tom K
  • 145
  • 1
  • 13

1 Answers1

0

Linux uses \n for a new-line, Windows \r\n and old Macs \r.

So essentially, what solved it for me was

select * from table where regexp_matches(bad_column, ['\r\n']);

then I could run

delete from table where id in (select id from table where regexp_matches(bad_column, ['\r\n']));

Tom K
  • 145
  • 1
  • 13