0

I have created an event hub triggered function app, which would received from one event hub and send message/data to another event hub using 

public static async Task Run(
    [EventHubTrigger("source", Connection = "EventHubConnectionAppSetting")] EventData[] events,
    [EventHub("dest-1", Connection = "EventHubConnectionAppSetting")]IAsyncCollector<string> outputEvents,
    ILogger log)

however, right now I would like to publish the same message to one or more additional event hubs(e.g, dest-2, dest-3, etc) so all my consumer event hubs(dest-1, dest-2, dest-3) could consume same message async. Is there anyway to achieve that approach with Azure Event hub triggered function app?

Drex
  • 3,346
  • 9
  • 33
  • 58

1 Answers1

1

If you don't want to create multiple async collectors then You need to use custom binder with IBinder class. Like below.

public static async Task Run(
    [EventHubTrigger("source", Connection = "EventHubConnectionAppSetting")] EventData[] events,
    IBinder binder,
    ILogger log) {

    // custom binding here (exmaple)
    var collector = await binder.BindAsync<IAsyncCollector<string>>(
                    new EventHubAttribute(... params to event hub here...));

    var message = ...
    await collector.AddAsync(message);

}

I typed this from my phone so sorry if there are typos ;) But in general this is a way to go.

Adam Marczak
  • 2,257
  • 9
  • 20
  • actually I would like to create multiple async collector. I've tried your solution but it's like single collector every time, correct? I implemented as ```var collector1 = await binder.BindAsync>(new EventHubAttribute("dest-1"));var collector2 = await binder.BindAsync>(new EventHubAttribute("dest-2"));var collector3 = await binder.BindAsync>(new EventHubAttribute("dest-3"));await collector1.AddAsync(message);await collector2.AddAsync(message);await collector3.AddAsync(message);```, but that looks wired somehow.. – Drex Aug 06 '19 at 20:58
  • I thought you wanted dynamic amount of collectors. If they are static, can't you just add multiple bindings? `[EventHub("dest-1", Connection = "EventHubConnectionAppSetting")]IAsyncCollector outputEvents1, [EventHub("dest-2", Connection = "EventHubConnectionAppSetting")]IAsyncCollector outputEvents2` – Adam Marczak Aug 06 '19 at 22:01
  • Got it! I understand now what you mean dynamic collectors, and yes that'd probably what I am looking for, but if I do `outputEvents1.AddAsync(message1); outputEvents2.AddAsync(message2);`, will it be async/parellel jobs for both outputEvent1 and outputEvent2? – Drex Aug 07 '19 at 13:27
  • It will be async but not parallel. This is not how async works, I would advise to read how to parallelize function execution but personally I like to do 'fan-out' approach where I put those intro azure queue like `{ hubName: 'dest-1', message: 'my message' }` and `{ hubName: 'dest-2', message: 'my message' }` so that two parallel functions will be executed. – Adam Marczak Aug 07 '19 at 14:17
  • thank you! Yes and 'fanout' was something I should mention earlier! I've done the fanout scenario with RabbitMQ before but not sure if Azure Event Hub or some other tech could do the same job, so the azure queue would do the similar job as 'fanout' right? I would definitely like into that later on, thank you very much for your advice! – Drex Aug 07 '19 at 15:11
  • Storage Queue is one of many services for fan out scenario. There is also pretty cool fan out example using durable functions instead of queue. https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-concepts#fan-in-out – Adam Marczak Aug 07 '19 at 15:17
  • thank you for the helpful link! I've read and tried myself on that, so based on the documentation example of fan out and fan back, it's like the F1 is holding there in Azure while it waits for all messages/events finished with F2, then grouping the result and sent to F3. However, from what I know, functions will have a lifetime around 10 mins, means if while F1 is waiting for results from 3 instances of F2 more than 10 mins, it will fail. My thought of fixing this issue is not to use the fan back in F1, instead directly send result from F2 to F3 in each instance, will that work? – Drex Aug 09 '19 at 14:58
  • Also I learned something with service bus topic which seems can perform the fan out option as well, so what would be the difference between using the service bus topic vs queue storage? Is it because queue storage is better working with functions? – Drex Aug 09 '19 at 15:06
  • Both work fine with functions. Service bus is just big service for big solutions, but it is also more expensive. Read this to check when to use which https://learn.microsoft.com/en-us/azure/event-grid/compare-messaging-services – Adam Marczak Aug 09 '19 at 15:59