Is it possible to have single error capture plugin for a pipeline? And how to route error records to that plugin from all other plugins?
Asked
Active
Viewed 107 times
1
-
Currently no. Each path in the pipeline needs to have a plugin that can emit error records and only those records can be captured in a pipeline. To understand more, what is it you are trying to do? And why are you looking for one error record capture plugin per pipeline? – Ajai Jan 25 '21 at 17:59
-
Primary usecase is to have error capture at least for each wrangler at the start of the path in pipelines. Rather than having multiple ones, one single saves effort and keeps pipeline much more organized. – adiideas Jan 27 '21 at 18:49
-
This is not possible right now. In the plugin architecture in CDF, each individual plugin has to be orchestrated to emit error records and not all plugins emit them. For each plugin emitting the error records, the schema for the error includes in the incoming schema for the plugin emitting the error + error code and error message. Don't believe having a generic error plugin for a pipeline would be able to capture this. – Ajai Jan 28 '21 at 05:05