0

Ok, this is a pretty unscientific question. Sorry in advance. Any of you using windows azure (and preferably SQL Azure database) seeing really random performance issues after you publish (or upgrade) code?

What I'm seeing could even be up to 24 hours after the publish. It's just general latency. http(s) requests take an extreme amount of time to return (30sec - 1 min), but the same request could return in milliseconds the next time it's called.

It seems very random. When I publish code to azure what all needs to change? Load balancing, DNS cache, IP addresses, etc... could the propagation of all of these network layer changes be causing my issue?

BTW, in almost all cases I do an upgrade of the staging environment and then VIP swap to production.

Taylor Bird
  • 7,767
  • 1
  • 25
  • 31
BZink
  • 7,687
  • 10
  • 37
  • 55

1 Answers1

1

Is this mainly with your web roles? Are they running .NET code? (ASP.NET?, WCF?, etc)

If so, it is possible you are dealing with normal .NET JIT delay on an IIS application that has been recycled due to activity. The symptom sounds just like when you have an ASP.NET app on-site and don't hit it for some amount of time. The IIS worker gets recycled and the runtime will not JIT-compile your app until a new HTTP request comes in. This creates a situation where that first request can take "forever" but every request that comes in in the next few minutes is as "instant" as you'd expect from your code.

This isn't Azure specific, but you could be dealing with an IIS environment different from what you run on-site (in regards to default app pool recycle settings/warmup settings after recycle).

edit with suggestions

If you suspect it's related to warm-up, there are a few solutions.

The best (unless you need otherwise) is to managed the root cause, which is IIS recycling the app pool. By default this happens on a timer or on a request count (not sure which in Azure). You can override when this occurs in your web.config with the <recycling></recycling> element. IIS.net has the best description of these settings. Take a look at:

http://www.iis.net/ConfigReference/system.applicationHost/applicationPools/add/recycling

and see if that helps. I would recommend just doing a timed recycle during a time period where you aren't being hit (middle of the night, for example)

Another option is to ensure traffic continuously hits the site. with some sort of polling software. "uptime/status" monitors like pingdom are great for this. This is a hack-approach, but I've used it before in odd scenarios. (woudln't recommend)

If that doesn't work due to special startup requirements (which is sounds like you don't have) IIS has a warm-up module, and C# 4.0 has warm-up classes which can help. Again, there are more to ensure you control what happens during startup instead of when startup occurs.

Taylor Bird
  • 7,767
  • 1
  • 25
  • 31
  • That's very interesting. Yes, this is in the ASP.net web roles. Given that we have multiple load balanced instances too, that would explain the additional randomness of the issue. One request may return instantly but the next request would take "forever" and then everything goes back to normal. If you don't mind... point me in the right direction on dealing with this. Thanks! – BZink Jan 26 '11 at 16:04