CURL request to kafka-rest

Question

I have executed HTTP POST on kafka-rest using curl call. one request is successful but another request (with different json) is returned 422 error ({"error_code":422,"message":"Unrecognized field: receiver"}).

Working request

    curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" -H "Accept: application/vnd.kafka.v2+json" --data '{"records":[{"value":{"foo":"bar"}}]}' "http://localhost:8082/topics/jsontest"

Outout

{"offsets":[{"partition":0,"offset":0,"error_code":null,"error":null}],"key_schema_id":null,"value_schema_id":null}

Not working request

 curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" -H "Accept: application/vnd.kafka.v2+json" --data @alert-request.json "http://localhost:8082/topics/jsontest"

Output

{"error_code":422,"message":"Unrecognized field: receiver"}

alert-request.json

{
  "receiver": "email-logs",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "CPUUsageAbove20%",
        "instance": "node-exporter:9100",
        "job": "node-exporter",
        "monitor": "my-project",
        "severity": "warn",
        "team": "raptors"
      },
      "annotations": {
        "dashboard": "www.prometheus.io",
        "description": "CPU usage on node-exporter:9100 has reached 60"
      },
      "startsAt": "2020-04-30T08:03:17.309164516Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://0ab6d9955c65:9090/graph?g0.expr=60+%3E+job%3Anode_cpu_seconds%3Ausage+%3E+20\u0026g0.tab=1",
      "fingerprint": "9c558a8c20c2ba08"
    }
  ],
  "groupLabels": {
    "alertname": "CPUUsageAbove20%",
    "team": "raptors"
  },
  "commonLabels": {
    "alertname": "CPUUsageAbove20%",
    "instance": "node-exporter:9100",
    "job": "node-exporter",
    "monitor": "my-project",
    "severity": "warn",
    "team": "raptors"
  },
  "commonAnnotations": {
    "dashboard": "www.prometheus.io",
    "description": "CPU usage on node-exporter:9100 has reached 60"
  },
  "externalURL": "http://5493399b56dc:9093",
  "version": "4",
  "groupKey": "{}/{team=~\"^(?:(raptors|leafs))$\"}:{alertname=\"CPUUsageAbove20%\", team=\"raptors\"}"
}

I need to write above json to kafka using kafka-rest, and i use curl call to perform HTTP post on kafka-rest but it's returning error. how to successfully write to kafka using kafka-rest (using curl)?.

score 1 · Accepted Answer · answered May 10 '20 at 16:53

The error code in the response as "error_code":422 means that the request payload is incorrect.

According the API spec of kafka-rest, the one of the valid request payload structure is

{
  "records": [
    {
      "key": "recordKey",
      "value": "recordValue"
    }
  ]
}

So your alert-request.json file should be modified with the contents under the value field inside records

{
  "records": [
    {
      "key": "recordKey",
      "value": {
        "receiver": "email-logs",
        "status": "firing",
        "alerts": [
          {
            "status": "firing",
            "labels": {
              "alertname": "CPUUsageAbove20%",
              "instance": "node-exporter:9100",
              "job": "node-exporter",
              "monitor": "my-project",
              "severity": "warn",
              "team": "raptors"
            },
            "annotations": {
              "dashboard": "www.prometheus.io",
              "description": "CPU usage on node-exporter:9100 has reached 60"
            },
            "startsAt": "2020-04-30T08:03:17.309164516Z",
            "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "http://0ab6d9955c65:9090/graph?g0.expr=60+%3E+job%3Anode_cpu_seconds%3Ausage+%3E+20\u0026g0.tab=1",
            "fingerprint": "9c558a8c20c2ba08"
          }
        ],
        "groupLabels": {
          "alertname": "CPUUsageAbove20%",
          "team": "raptors"
        },
        "commonLabels": {
          "alertname": "CPUUsageAbove20%",
          "instance": "node-exporter:9100",
          "job": "node-exporter",
          "monitor": "my-project",
          "severity": "warn",
          "team": "raptors"
        },
        "commonAnnotations": {
          "dashboard": "www.prometheus.io",
          "description": "CPU usage on node-exporter:9100 has reached 60"
        },
        "externalURL": "http://5493399b56dc:9093",
        "version": "4",
        "groupKey": "{}/{team=~\"^(?:(raptors|leafs))$\"}:{alertname=\"CPUUsageAbove20%\", team=\"raptors\"}"
      }
    }
  ]
}

Do note that the key field is optional and you can provide any unique key that suits your use-case.

is it possible to ommit "records" and "value" in the json. like adding json schema to kafka schema proxy?. — Abdul Rahuman Seeni Mohamed, May 10 '20 at 18:19
No @AbdulRahumanSeeniMohamed. If you want to use the kafka-rest API to send a message payload to a kafka topic, that is their contract. You can only use an API as per the contract. — Madhu Bhat, May 11 '20 at 09:07
Yes thanks @Madhu Bhatt, In order to use kafka-rest, i need to follow the KAFKA-REST json format. All i can do is create a small rest service in Python/Java/GO, format the JSON accordingly and hen send it to kafka-rest. — Abdul Rahuman Seeni Mohamed, May 25 '20 at 11:48

CURL request to kafka-rest

1 Answers1