1

We have been running a production application on Azure for almost 3 years now that relies on web-hook notifications of incoming messages arriving on Office 365 mailboxes. Within the past day or so, we have noticed a significant percentage of messages for which we are not getting web hook notifications. All of our mailbox subscriptions are current and correct - a given mailbox may get 30 messages in an hour, and we may only see notifications from 25. We need to get notifications for 100% of them.

I want to emphasize that we have no reason to believe this is a programming issue - this portion of the application has not changed in months, and most of the time is working just fine. None-the-less, we have been told by Azure Support that the only means of getting production support for this issue is here at SO.

dmulter
  • 2,608
  • 3
  • 15
  • 24
Chuck
  • 55
  • 5

2 Answers2

0

If the Incoming WebHooks app has certain permissions, the caller of the web hook URL may override the configured channel, name, and icon. This is why messages posted via a specific Incoming WebHooks configuration may be using completely different settings!

Dynatrace is using these override features for alert messages, so alerts will always go to the channel configured on Dynatrace side.

Reference from:

Settings of Incoming WebHooks configuration are ignored

Alina Li
  • 884
  • 1
  • 6
  • 5
0

The following answer was posted and later deleted - I don't understand why.

"Sorry about the issues you experienced with Microsoft Graph webhook notifications.

Here is what happened:

We experienced a capacity issue that resulted in a delayed deliveries for a subset of notifications. Some subscriptions were affected more than others, with some subscribers seeing a significant drop in notifications delivered. The time period for this issue was between 2AM on 12-12-2018 and 6PM 12-13-2018 (PST).

What we are going to do:

  1. We are investigating the root cause and will take steps to prevent this from occurring in the future.

  2. We recognize that our outage communications for Graph notifications are lacking. We are working on improving our comms with the goal of pro-actively publishing service degradation info to our customers as soon as we realize they occur.

We appreciate your usage of Graph and our webhook notification framework. Sorry about the trouble!"

Chuck
  • 55
  • 5
  • We are continuing to experience missing notifications - 15 yesterday, 23 the day before. While this is an improvement over last week, this is still unacceptable for our application to perform effectively. Can I get an update on when a compete solution will be available? – Chuck Dec 20 '18 at 18:17
  • After months of working with Microsoft to attempt to resolve this issue, we were finally told that due to architectural issues in Exchange, Office 365 is unable to provide any guarantees on the delivery of Web Hook notifications. As a result, we are rearchitecting our solution to eliminate relying on notifications. – Chuck Aug 19 '19 at 20:41
  • Hi @Chuck. I appreciate that this was quite a while ago, however I am now coming across the same problem. I have a Logic App configured as the notificationUrl for my subscription. The I now have a situation where calls sometimes do not appear to hit the notificationUrl or are duplicated. I've double-checked the configured subscription (only one) and all seems fine there. I cannot work out why this would be happening. Is this similar to the problem you were having with the webhooks? Thanks – huwparry22 May 11 '20 at 12:32
  • @huwparry22 I would not be surprised if your issue is the same as mine. What I was told by Microsoft is that underneath MsGraph, Office 365, and all the other various API layers, it is Exchange that is fundamentally unable to provide any reliable notification of events. They told me that notifications should never be lost, but could be delayed for hours. But I gave them proof that notifications could go missing for over 24 hours - which is as good as lost in my book. The response I got was that the problem with Exchange has been well understood for many years and there are no plan to fix it. – Chuck May 11 '20 at 16:26
  • thanks for your reply. Glad to hear it's not just me who is experiencing this problem! – huwparry22 May 12 '20 at 08:49