0

I'm trying to send 2 custom events to kafka, the idea is to send each event to its topic, but at the moment I receive both events in both topics.

I have configured 2 mappings with different schemas and 2 sinks (one for each mapping).

here is the divolte-collector.conf:

divolte {
  global {
    kafka {
      enabled = true
      threads = 2
      buffer_size = 1048576

      producer = {
        bootstrap.servers = "localhost:9092"
        acks = 1
        retries = 0
        compression.type = lz4
        max.in.flight.requests.per.connection = 1
      }
    }
  }

  mappings {
    a_mapping = {
        schema_file = "path-to/conf/MyEventRecord.avsc"
        mapping_script_file = "path-to/conf/mapping.groovy"
        sources = [browser]
        sinks = [kafka1]
    }
    b_mapping = {
        schema_file = "path-to/conf/MyEventRecord2.avsc"
        mapping_script_file = "path-to/conf/mapping2.groovy"
        sources = [browser]
        sinks = [kafka2]
    }
  }

  sinks {
    kafka1 {
      type = kafka

      topic = topic-1
    }
    kafka2 {
      type = kafka

      topic = topic-2
    }
  }
}

my groovy file:

mapping {
  map eventParameter('js_error_msg') onto 'js_error_msg'
  map eventParameter('js_error_level') onto 'js_error_level'
  map eventParameter('js_url') onto 'js_url'
  map eventParameter('js_line') onto 'js_line'
  map eventParameter('js_column') onto 'js_column'
  map eventParameter('js_is_error_caught') onto 'js_is_error_caught'
}

my avro file:

{
  "name": "tracking",
  "type": "record",
  "fields": [
    { "name": "js_error_msg", "type": "string" },
    { "name": "js_error_level", "type": "string" },
    { "name": "js_url", "type": "string" },
    { "name": "js_line", "type": "string" },
    { "name": "js_column", "type": "string" },
    { "name": "js_is_error_caught", "type": "string" }
  ]
}
Soufiane Odf
  • 1,054
  • 2
  • 9
  • 23
  • 1
    You've configured both sources as the browser, so the browser is linked to both sinks without any kind of filtering... But can you share your Groovy files? – OneCricketeer Aug 21 '21 at 16:20
  • yes I have the same source, this is the difference, but I need to channel by type for example or something like that, is it possible to do that ? about groovy file, I don't have problem with it because I receive the data, and I don't think it has something to do with that, or I'm wrong ? – Soufiane Odf Aug 21 '21 at 16:23
  • I have published the Groovy and avro file, do you think I can use the type field in avro file to channel the events ? – Soufiane Odf Aug 21 '21 at 16:33
  • 1
    You've only shown one pair of files, but the type is a record in both cases, I assume, so no, not on the type... You could probably use the url, though. Specifically, look into using exit or stop mapping functions when the url doesn't match what you want http://divolte-releases.s3-website-eu-west-1.amazonaws.com/divolte-collector/0.9.0/userdoc/html/mapping_reference.html – OneCricketeer Aug 21 '21 at 19:22
  • I see what you're saying, but this would block the event or let it pass to both topics, because the weird thing is that if only one mapping gets the event, then he is going to pass it to both topics. – Soufiane Odf Aug 21 '21 at 19:47
  • 1
    All mappings are going to get all events because that's what the source property does. It's up to the individual mapping files to filter/drop events, which prevent them from going on to the sinks – OneCricketeer Aug 21 '21 at 21:23
  • 1
    Also, I assume you saw this issue? https://github.com/divolte/divolte-collector/issues/483 – OneCricketeer Aug 21 '21 at 21:35
  • Yes I have already seen that issue, and about the filtering thing, I have removed the second mappping and kept only the first one, which has only kafka1 as a sink, and in the sinks part, I kept the two sinks, kafka1, and kafka2 which I'm not pointing on at that moment in the mapping, but guess what, I get also the event in kafka2, which is very weird. – Soufiane Odf Aug 21 '21 at 22:41
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/236272/discussion-between-soufiane-odf-and-onecricketeer). – Soufiane Odf Aug 21 '21 at 23:27

0 Answers0