I'm trying to develop a plugin similar to the HTTP sink plugin that comes by default in CDAP, I need to buffer incoming data and send out via an HTTP Post method whenever the buffer becomes full.
I'm using Cloud Data Fusion version 6.7.2 in GCP.
I referred to the http sink plugin code HttpSink, here they are making use of the destroy lifecycle method to send the remaining items in the batch.
While running the pipeline, I found that the destroy method doesn't get invoked at all and the items remaining in the buffer are never sent out. The same behaviour is there in the official HTTP sink plugin as well.
The official [documentation (https://cdap.atlassian.net/wiki/spaces/DOCS/pages/480412201/Creating+a+Plugin) clearly mentions that the destroy method will be called after all calls to the transform method are completed.
I don't understand why this doesn't get called. Can anybody shed some light on this?
I tried creating a plain old Batch Sink class that simply logs the data. I did the same with all the other lifecycle methods to see the order in which they were called. I expected the destroy method to get called after all the calls to the transform method were made. But I couldn't find the logs I had written in the destroy method.
