1

I'm trying to post data(payload) from BitBucket to my Kafka instance in case of any BitBucket events (like Pushed, Forked, PR creating, Addition/Modification/Deletion of PR comments etc.)

So, I've setup Kafka rest-proxy but I can't send data from BitBucket to Kafka as the Content-Type required to post data using the Kafka REST proxy is application/vnd.kafka.json.v2+json and BitBucket's payload has an header set as application/json (I'm using the default BitBucket webhook configured at the Repository level).

I've configured the Webhook at the repository level is this way: enter image description here

This is the request which is being triggered from BitBucket's Webhook:

enter image description here

And the response I get from Kafka's REST Proxy is this:

enter image description here

I've thought about the following solutions:

  1. I can write a simple web server which acts as a mediator between the two systems (BitBucket and Kafka). The server can act as an intermediate proxy and grab incoming requests from BitBucket's webhook and post it to Kafka REST proxy using the appropriate header (application/vnd.kafka.json.v2+json).

    This solution works but the downsides are maintenance of another server (written by me). Also, it can have scalability woes as BitBucket starts triggering webhooks frequently (100s per second during peak hours as we've only one BitBucket instance and we've around 3500 developers).

  2. I can use a BitBucket plugin which works as a webhook and which allows me to configure and change headers on the fly. This is the solution which I'm thriving to achieve. Doesn't involve an extra server to be maintained as well.

Is there a BitBucket plugin, which I can install from the market place which will enable me to configure the header when the webhook is triggered?

P.S. - Apologies for writing such a long question. These two solutions are just from the top off my head. I won't be surprised if a better solution is available to integrate BitBucket with Kafka. Also, please do let me know if the question stands unclear. In that case, I'll try to clarify my question in a different manner.

Thanks

Aryak Sengupta
  • 1,727
  • 2
  • 18
  • 23

1 Answers1

0

You can use Pipedream to accept webhook requests from Bitbucket, modify the Content-Type header to application/vnd.kafka.json.v2+json, and forward the request to your Kafka REST proxy. Running pipelines like this on Pipedream is free.

I created an example pipeline that shows you how this works. It takes the HTTP payload sent from Bitbucket — $event.body — and uses a bit of Node.js code to forward the request on to another HTTP endpoint.

In my case I just added a RequestBin URL so you can see the body and headers being POSTed somewhere — check out an example request I sent here. Expand the Headers section of that request and notice the Content-Type header is correctly set to application/vnd.kafka.json.v2+json.

If you fork the pipeline above (just click the Fork button in the top-right), it should generate a custom URL specific to your pipeline. Add that URL as a new Bitbucket webhook URL tied to your repo, and you should start seeing new events come through.

This assumes your Kafka REST proxy URL is publicly-accessible. All the code for Pipedream pipelines is public, so if you want to keep your Kafka REST Proxy endpoint private, you can create an environment variable and reference that in the Node.js code, replacing the RequestBin URL with the value of that environment variable.

For example, I might create an environment variable called KAFKA_REST_ENDPOINT and replace:

url: 'https://entygbild98b.x.pipedream.net'

with

url: process.env.KAFKA_REST_ENDPOINT

Let me know if this helps or if you have any other questions.

Dylan Sather
  • 30
  • 6