How can we configure the Kafka connector in IBM DataStage to continuously pick up new records at a rate of 28 records per second and trigger the data pipeline?
Asked
Active
Viewed 51 times
0
-
There's no point in setting the reading rate to a fixed speed. I don't know if they fixed it yet, but the last time I tried to get Kafka connector running, it wasn't able to read contiously without having the job stuck. I could only read less rows than there were available in the topic. If I tried to read 100 rows, but the topic only had 99, the connector wouldn't close to start the waive. I solved that with a shell script that checked how big the offset gap in the topic was to get the exact amount of rows I needed to fetch. Could have passed that int as param, but then directly used files. – Justus Kenklies Apr 12 '23 at 07:09