Druid with Kafka Ingestion: filtering data

Question

is it possible to filter data by dimension value during ingestion from Kafka to Druid?

e.g. Considering dimension: version, which might have values: v1, v2, v3 I would like to have only v2 loaded.

I realize it can be done using Spark/Flink/Kafka Streams, but maybe there is an out-of-the-box solution

I need this feature, too, but I've seen a few times that Druid people mentioned the ETL is not the scope of Druid. I think it's difficult to implement a general purpose ETL. — Kenji Noguchi, Mar 27 '17 at 17:31

score 3 · Accepted Answer · answered Oct 23 '18 at 15:16

You can do this with transformSpec during ingestion.
http://druid.io/docs/latest/ingestion/transform-spec.html

Per the documentation:

Transform specs allow Druid to filter and transform input data during ingestion.

Any query filters can be applied to this.

Example usage with NOT filter:

"transformSpec": {
  "filter": {
    "type": "and",
    "fields": [
      {
        "type": "not",
        "field": {
          "type": "selector",
          "dimension": "my_dimension",
          "value": "filter_me"
        }
      },
      {
        "type": "not",
        "field": {
          "type": "selector",
          "dimension": "my_dimension",
          "value": "filter_me_also"
        }
      }
    ]
  },
  "transforms": []
}

score 1 · Answer 2 · answered Mar 20 '17 at 21:03

1

Not possible from druid side you need to filter the data before hand.

answered Mar 20 '17 at 21:03

Slim Bouguerra

359
1
8

Outdated answer. – pcejrowski Oct 23 '18 at 16:02
I also tried native batch ingestion with task (firehose). its actually reindexing existing segments with filters. but didn't work with filters in tansformSpec. – shadee Sep 27 '19 at 17:40

Druid with Kafka Ingestion: filtering data

2 Answers2