Unable to load csv file in wrangler data fusion

Question

i am trying to import a file which has more than 100 columns and size of file is 4 mb.

after applying operation "parse-as-csv :body ',' true" . no data is showing

but if file size is < 2mb it is working.

please give any solution

Answer 1

This is a known issue with Wrangler, where the headers are not filtered after the first file. Saying that, in the first file, you have columns:

| 1 | 2 | 3 |

And in the second one, there is a mismatch where the columns are aligned like:

| 2 | 1 | 3 |

As a remediation, you can parse with the delimiter in the source and filter the header. Another possible approach that I can think of is to use Python to preprocess the data and align the headers.

Additionally, Data Fusion can import files up to 250 MB.