0

When I create a stream in the pipeline and it starts consuming of a topic, it does not appear in the active consmidores of my topic, what do I do so that it appears there?

No consumer appears

How I created the stream 

CREATE STREAM pipeline_kafka.summary_device_stream(pack json);
SELECT pipeline_kafka.consume_begin('topicSummaryDevice', 'pipeline_kafka.summary_device_stream', format := 'json');
miguno
  • 14,498
  • 3
  • 47
  • 63
luiz kava
  • 187
  • 1
  • 2
  • 14

1 Answers1

0

I'm not sure exactly where that value is getting pulled from in your attached screenshot, but I'm guessing it only applies to consumers that are storing offsets in Kafka.

You can store offsets in Kafka with pipeline_kafka by passing a group_id parameter to the consumer command, like so:

SELECT pipeline_kafka.consume_begin('topicSummaryDevice',
   'pipeline_kafka.summary_device_stream',
    group_id := 'consumer_group_id',
    format := 'json');
Derek Nelson
  • 178
  • 1
  • 5
  • grave error :( **SQL Error [42883]: ERROR: function pipeline_kafka.consume_begin(unknown, unknown, group_id => unknown, format => unknown) does not exist Dica: No function matches the given name and argument types. You might need to add explicit type casts. Posição: 8 org.postgresql.util.PSQLException: ERROR: function pipeline_kafka.consume_begin(unknown, unknown, group_id => unknown, format => unknown) does not exist Dica: No function matches the given name and argument types. You might need to add explicit type casts. Posição: 8** – luiz kava Apr 26 '17 at 21:52
  • What is the exact command you entered that caused that error? – Derek Nelson Apr 28 '17 at 15:53
  • this command ... `SELECT pipeline_kafka.consume_begin('topicNotificationProcess', 'pipeline_kafka.sensor_event_process_stream', format := 'json', group_id:= 'groupEventProcess'; batchsize := 4330, parallelism := 5, start_offset := -1);` – luiz kava May 01 '17 at 14:43
  • That command has a `;` after the `group_id` argument instead of a `,` which actually isn't even valid syntax. Other than that, you should verify that the `pipeline_kafka` extension is installed by checking the output of `\dx`. – Derek Nelson May 02 '17 at 15:54
  • I put it wrong here but I used the "," after group_id.... The extension is installed, the consumer works, it just does not appear in a group in the kafka. – luiz kava May 02 '17 at 16:42