I am receiving following error while executing Data Pipeline in GCP Cloud Data Fusion.
Spark program 'phase-1' failed with error: canCommit() is called for transaction
More information: So the Pipeline is responsible for lift'n'shift operation, loading on-prem oracle data into Google Bigquery via cloud data fusion. The pipeline gives this error on Intermittent basis, meaning, it works sometime (Manual run), but mostly failed on schedule run (sometimes works on schedule run as well).
As part of mitigation, I have set the following configuration item, but no luck.
*data.tx.timeout*
Thanks a lot in advance.
Regards, Vir
