Azure Event Grid advance filtering with Cloud Event Schema v1.0

Question

I'm using Event Grid for communicating two different microservices. For this purpose I have created a topic and a subscription:

Event Grid Topic with Cloud Event Schema v1.0
Event Grid Subscription with Cloud Event Schema v1.0

Events are sent and received correctly. However, I'm trying to add an advanced filtering based on the 'source' field of the CloudEvent as you can see in the image. Everything seems to be properly configured but I'm receiving all events in my subscription regardless the source in the event.

Any idea what could I miss here?

Roman Kiss · Answer 1 · 2019-11-24T11:25:19.287

In the CloudEventV01Schema, the source property consists of the topic and subject values with a delimiter character #, see the following:

source = topic#subject

Using an OperatorType = StringNotIn requires to match the full value of the key, such as in your case the source value.

Adding the topic and delimiter # to your subject (such as https://my-source.azurewebsites.net/) will fix your issue.

Example for the custom topic:

/subscriptions/yourSubscriptionId/resourceGroups/yourResourceGroup/providers/Microsoft.EventGrid/topics/yourTopic#https://my-source.azurewebsites.net/

Please, vote adding more operationTypes in the AdvancedFilters, where your issue can be handled very easy using an operationType = StringNotEndsWith.

Update:

In the CloudEventSchemaV1_0 is the source property represented by the AEG topic, see an example for storage producer. In your case such as a custom topic, the source will have a value of the custom topic resourceId.

In other words, there is a useless using an advantage filtering for source property, because the AEG Pub/Sub eventing model supports only subscribing to the one specific topic (one subscription to one topic).

Note, that using the event domain topic instead of the custom topic, the source property is represented as a topic in the single event domain where we can filtered a source (domain topic) for consuming or direct subscribing for this source (domain topic).

In conclusion, the current public preview version (api-version=2020-01-01-preview) doesn't support what you are expecting from the advanced filtering on the source property.

Update2:

I have created an issue related to consume a CloudEvents v 1.0 by the AEG custom topic endpoint.

Mean time, the following workaround can be used for your solution:

 Event Grid Topic with CustomInputSchema
 Event Grid Subscription with CustomInputSchema (or CloudEventSchemaV1_0)

Example:

Create a custom topic using the REST API with the following payload:

{
  "location":"westus",
    "tags":{
      "tag1":"abcd",
      "tag2":"ABCD"
      },
  "properties":{
    "inputSchema":"CustomEventSchema",
    "inputSchemaMapping":{
    "properties":{
      "id":{
        "sourceField":null
      },
      "topic":{
        "sourceField":null
      },
      "eventTime":{
        "sourceField":null
      },
      "eventType":{
        "sourceField":null,
        "defaultValue":"DefaultEventType"
        },
      "subject":{
        "sourceField":null,
        "defaultValue":"DefaultSubject"
      },
      "dataVersion":{
        "sourceField":null,
        "defaultValue":"1.0"
      }
    },
    "inputSchemaMappingType":"Json"
    }
  }
}

The above custom mapping will allow to bypass any event schema from the custom topic endpoint to the subscriber. If the subscriber is declared for delivery schema with a CloudEvents v 1.0, then the event is wrapped by the CloudEventSchemaV1_0 envelope.

In this workaround the event message must be in the array, see the following sample:

[
  {
    "id":"00000000-0000-0000-0000-000000000000",
    "source":"https://my-source.azerwebsites.net/",
    "specversion":"1.0",
    "type":"recordInserted",
    "dataschema":"#1.0",
    "subject":"/myapp/vehicles/motorcycles",
    "time":"2019-11-23T16:43:22.5111403Z",
    "data":{
      "make":"Ducati",
      "model":"Monster"
    }
  }
]

Note, that this custom input schema wrapping by array should be handled the same as CloudEvents such as for a single event as an JObject and for batching as an JArray, see my reported issue here.

As you can see the above event message is a CloudEvents v 1.0 message and in this workaround you can use your advanced filtering on the source attribute.

Thanks for the update. As a workaround I'm doing the filtering in the subscription endpoint directly in code. Easier than changing the deploying templates. — DavidGSola, Nov 25 '19 at 09:53
That's true, when you are using a *custom topic* for very small eventing model. For application with more topics, partitioning, etc., the event domains is more preferable model (https://learn.microsoft.com/en-us/azure/event-grid/event-domains) and using a workaround in the subscriber code will not help, the source attribute of the CloudEvents must be a domain topic (e.g. *abcd*), that's mapping by the AEG team, otherwise the publisher will receive a BadRequest error code. I hope, that the Microsoft AEG team will change it to follow a CloadEvents v 1.0 support on the consuming side. — Roman Kiss, Nov 25 '19 at 10:32

Azure Event Grid advance filtering with Cloud Event Schema v1.0

1 Answers1

Linked