0

in my application I have a scenario where there are events are continuously generated, each event is associated with a type.

For every event multiple sub actions depending on the type need to be performed which are totally independent.

So for this scenario solution which I thought of is to use Queue along with topics as below

Event generator ---> Queue -----> Orchestrator --------> topics <----- Listeners

to implement the above there is a need to have an orchestrator which gets the events from the queue and apply some logic and insert into topics .

Can any one suggest any good component for the orchestrator on azure ?

Orchestrator should be able to handle load(Spikes are expected) - As we are using queues processing an event need not be a realtime it can be a delayed based on the load.

Mahesh Gupta
  • 457
  • 1
  • 6
  • 18

1 Answers1

1

I have two options for you.

I. Service Bus forwarding

If the rules of forwarding are straightforward (e.g. based on a property value), you can setup this with Service Bus filterer subscriptions and forwarding.

You create an ingress topic which will get all the messages. Then you create a subscription for this topic per each type of message, and you apply a filter for that subscription to filter out all the other types. You then setup the forward rule to send the messages to the appropriate outgoing topic:

var description = new SubscriptionDescription("IngressTopic", "Type1");
description.ForwardTo = "SinkType1Topic";
SqlFilter filter = new SqlFilter("Type = 1");
namespaceManager.CreateSubscription(description, filter);

where Type is the property in your message. You repeat this for each type.

II. Azure Function

If you need to run your custom code to determine which outgoing topic the message belongs to, you can run this custom code as Azure Function. Just make an ingress queue, and create a Function which will listen to this queue, receive the message and forward it to the appropriate outgoing topic:

public static void Run(MyMessage message, IBinder binder)
{
    string outgoingTopic = message.CalculateOutgoingTopic();
    var attribute = new ServiceBusAttribute(outgoingTopic);
    var collector = binder.Bind<ICollector<MyMessage>>(attribute);
    collector.Add(message);
}

Azure Functions are good for handling spiky load.

Mikhail Shilkov
  • 34,128
  • 3
  • 68
  • 107
  • hi mikhail.. orchestration can include transformation also, so forwarding seems to be out of reach. Functions seems to be a good option but pricing may go up as it has unlimited resources. Looking for a solution where cost can be limited and controlled. – Mahesh Gupta Jun 14 '17 at 09:59
  • @MaheshGupta You can run Functions on fixed App Service Plan, thus paying predefined monthly fee depending on instance size and count. You will loose the elasticity of Consumption Plan though, but that's your decision to make. – Mikhail Shilkov Jun 14 '17 at 10:18
  • If I run the functions on a fixed app service plan will it survive the load spikes? will the concurrent executions happen only when resources(memory and cpu) are available? will it be automatically handled as in Queue client library? – Mahesh Gupta Jun 14 '17 at 12:12
  • @MaheshGupta Yes, you'll just get some backlog in the queue, if it won't be able to keep up. – Mikhail Shilkov Jun 14 '17 at 12:26