EWS - ServerBusyException?

Question

After our company upgraded from Exchange 2010 to Exchange 2013, I found a really strange quirk. I'm using the EWS managed API to create a streaming subscription to some public folders. I have some older windows services still running that are happily subscribed, reading new mail, and stuff like that. I have my new project that will eventually replace these old windows services, and while we were on Exchange 2010 it could create the same streaming subscriptions to the same public folders while the legacy windows services were still running. Post upgrade, I am now getting a ServerBusyException when I try to open the StreamingSubscriptionConnection. If I disable all the legacy windows services, I can open the connection fine in my new project.

Here is how I'm subscribing, nothing too fancy:

    private StreamingSubscriptionConnection CreateStreamingSubscription(ExchangeService service, StreamingSubscription subscription)
    {
        var connection = new StreamingSubscriptionConnection(service, 30);
        connection.AddSubscription(subscription);
        connection.OnNotificationEvent += connection_OnNotificationEvent;
        connection.OnSubscriptionError += connection_OnSubscriptionError;
        connection.OnDisconnect += connection_OnDisconnect;

        try
        {
            connection.Open(); // <-- boom!
        }
        catch (ServerBusyException ex)
        {
            Log.Error("Connection to Exchange refused. Server is too busy.", ex);
            throw; // <-- this is the exception, it is specific
        }
        catch (Exception ex)
        {
            Log.Error("Unknown error connection to Exchange.", ex);
            throw;
        }


        return connection;
    }

After looking that MSDN, there is a property called BackOffMilliseconds, but it is coming back as 0 indicating that Exchange isn't actually telling me to try again later. I logged into the Exchange server to check throttling policies, and the default global policy is applied. Every parameter that has the word 'concurrency' in it is over 10, so I don't think it is a throttling policy issue. For what it's worth, my legacy windows services and this new project are all using the same service account to do their thing.

I also looked at how the service connection is made, and there is no change whether it is set to Exchange2013 or Exchange2010_SP1:

    private ExchangeService CreateExchangeService()
    {
        ServicePointManager.ServerCertificateValidationCallback = CertificateValidationCallBack;
        var service = new ExchangeService(ExchangeVersion.Exchange2013);
        service.Credentials = new WebCredentials(ExchangeUser, ExchangePassword, ExchangeDomain);
        service.AutodiscoverUrl(ExchangeEmail, RedirectionUrlValidationCallback);
        return service;
    }

Last I checked Fiddler to see what was happening during these failed subscriptions, and I'm seeing an unhelpful HTTP 500 with the same server busy, try later message.

Any ideas on what changed in the EWS API in Exchange 2013 vs 2010SP2?

score 4 · Accepted Answer · answered Jul 06 '14 at 03:31

4

Seems like you are hitting the default hanging connection limit which is 3 for on-premise Exchange 2013 server. You can override it in web.config as mentioned here. Basically add following node under appSettings xml node in web.config (it sets it to 10) and restart MSExchangeServicesApppool:

add key="HangingConnectionLimit" value="10" (append xml tags)

web.config for EWS app pool can be found at: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\exchweb\ews. It may vary depending on Exchange Installation directory.

Also note that this change needs to be made on Mailbox Server(s) where users active database exists.

answered Jul 06 '14 at 03:31

atul.goyal

126
3

This sounds like what I am hitting, even though the hanging connection is for a different error code. Before I modify the web.config on Exchange, is this something that can be set via Powershell for the default throttling policy? – Bill Sambrone Jul 06 '14 at 21:42
No, I don't think it is exposed via Powershell. What error code are you hitting? – atul.goyal Jul 07 '14 at 17:20
The exception type of ServerBusyException throws immediately when I try to try to call the .Open() method of StreamingSubscriptionConnection. – Bill Sambrone Jul 07 '14 at 18:31
Yeah, then you are likely hitting this same limit. – atul.goyal Jul 07 '14 at 19:43

score 2 · Answer 2 · answered Jul 03 '14 at 21:33

Although there are no changes to the EWS API in terms of subscriptions - Exchange itself changed where the subscriptions are maintained.

In Exchange 2010 the subscriptions were stored on the CAS, in 2013 they're stored on the Mailbox server.

Based on the exception, I think you are indeed getting throttled. There is a set of recommendations for maintaining affinity with the mailbox server in 2013 and avoiding throttling in this topic: How to: Maintain affinity between a group of subscriptions and the Mailbox server in Exchange. Basically, you group subscriptions and use X-AnchorMailbox header to route the requests.

And there are specifics about why you might be seeing the ErrorServerBusy Exception in this topic: Handling notification-related errors in EWS in Exchange.

Matt Stehle also blogged about it here: Changes in Managing Affinity for EWS Subscriptions….

EWS - ServerBusyException?

2 Answers2