How do I pass multiple CSVs with custom delimiter to a great_expetctation checkpoint

Question

I am trying to run great_expectation checkpoint on 10 CSV files with "|" delimiter. Currently, I have to specify this all in a YAML file and that after converting my files from "|" delimiter to ",".

How can run this for multiple files without specifying it in a YAML and that too with a custom delimiter?

Here's my configuration for the checkpoint.

Answer 1

The below link can be useful, it works for me

https://greatexpectations.io/blog/specify-nonstandard-delimiters-for-csvs-in-great-expectations

batch_request = BatchRequest(
  datasource_name=f"datasource_name",
  data_connector_name=f"data_connector_name",
  data_asset_name=f"data_asset_name",
  batch_spec_passthrough= {"reader_options": {"sep": "|","header": True}}
)