druid kafka indexing service setup

Question

I followed the docs and edited:

druid-0.9.2/conf/druid/_common/common.runtime.properties

and added:

"druid-kafka-indexing-service"

to the druid.extensions.loadList and restarted all druid services: middlemanager, overlord, coordinator, broker, historical

I ran:

curl -X 'POST' -H 'Content-Type:application/json' -d @kafka_connect/script.json druid_server:8090/druid/indexer/v1/task

but got:

{"error":"Could not resolve type id 'kafka' into a subtype of [simple type, class io.druid.indexing.common.task.Task]\n at [Source: HttpInputOverHTTP@4c467f1c; line: 1, column: 4]"}

The input json has:

{
  "type": "kafka",
  "dataSchema": {
    "dataSource": "sensors-kafka",
    "parser": {
      "type": "string",
      "parseSpec": {
        "format": "json",
        "timestampSpec": {
          "column": "timestamp",
          "format": "auto"
        },
        "dimensionsSpec": {
          "dimensions": ["machine", "key"],
          "dimensionExclusions": [
            "timestamp",
            "value"
          ]
        }
      }
    },
    "metricsSpec": [
      {
        "name": "count",
        "type": "count"
      },
      {
        "name": "value_sum",
        "fieldName": "value",
        "type": "doubleSum"
      },
      {
        "name": "value_min",
        "fieldName": "value",
        "type": "doubleMin"
      },
      {
        "name": "value_max",
        "fieldName": "value",
        "type": "doubleMax"
      }
    ],
    "granularitySpec": {
      "type": "uniform",
      "segmentGranularity": "HOUR",
      "queryGranularity": "NONE"
    }
  },
  "tuningConfig": {
    "type": "kafka",
    "maxRowsPerSegment": 5000000
  },
  "ioConfig": {
    "topic": "sensor",
    "consumerProperties": {
      "bootstrap.servers": "kafka_server:2181"
    },
    "taskCount": 1,
    "replicas": 1,
    "taskDuration": "PT1H"
  }
}

Any idea what I did wrong? According to the doc: http://druid.io/docs/0.9.2-rc3/development/extensions-core/kafka-ingestion.html, the type is kafka?

Is there a way to check that the extension was loaded properly or do I have to specify the extension in each component's runtime.properties?

score 3 · Answer 1 · answered Jul 04 '18 at 11:18

I meet the similar problems,and I fixed it by modify the file of "conf/druid/_common/common.runtime.properties" by add "druid-kafka-indexing-service" to druid.extensions.loadList,and now its show like this:

druid.extensions.loadList=["druid-parser-route", "mysql-metadata-storage", "druid-kafka-indexing-service"]

Hope can help anyone else

score 2 · Answer 2 · answered Feb 24 '17 at 21:41

2

The supervisors json specs are to be sent on this endpoint on the overlord /druid/indexer/v1/supervisor

curl -X POST -H 'Content-Type: application/json' -d @kafka_connect/script.json http://druid_server:8090/druid/indexer/v1/supervisor

answered Feb 24 '17 at 21:41

Pierre Lacave

2,608
2
19
28

If the error changed to something like Kafka cannot be resolved by overlord, I would double check that extensions are correctly loaded, it is printed at overlord startup – Pierre Lacave Feb 27 '17 at 07:35
1

okay worked it out. I was using the quickstart guide but the extensions documentation refers to `conf` not `conf-quickstart` which I was using the start the services... Now I got an `error 500` instead but that is because the data source has been coded to send out `timestamp` as epoch not `YYYY-MM-DD` etc – KillerSnail Feb 27 '17 at 07:39

score 0 · Answer 3 · answered Feb 06 '21 at 00:44

if using dockerized apache/druid, you need to set

druid.extensions.loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-kafka-indexing-service"]

in file /opt/druid/conf/druid/cluster/_common/common.runtime.properties

druid kafka indexing service setup

3 Answers3