1

I'm writing a high-ish volume web service in C# running in 64-bit IIS on Win 2k8 (.NET 4.5) that works with XML payloads and does a variety of operations on small and large objects (where the large objects are mainly strings, some over 85k (so going onto the LOH)). Requests are stateless, and memory usage remains steady over time. Lots of memory is being allocated and released per request, no memory appears to be being leaked.

Operating at a maximum of 25 transactions per second, with an average call lasting 5s, it's spending 40-60% of it's time in GC according to two profiling tools, and perfmon shows a steady 20 G0 and G1 collections over 5 seconds, and 15 G2 collections over 5 seconds - meaning lots of (we think) premature promtion into G2 for data that we'd expect to stay in G0. Everything I read indicates this is very excessive. We expect that the system should be able to perform at a higher throughput than 25 tps and assume the GC activity is preventing this.

The machines serving the requests have lots of memory - 16GB - and the application, under load, consumes at most 1GB when under load for an hour. I understand that a bigger heap won't necessarily make things better, but there is spare memory.

I appreciate this is light on specifics (will try to recreate the conditions with a trivial application if time permits) - but can anyone explain why we see so much G2 GC activity? Should I be focusing on the LOH? People keep telling me that the CLR's GC "adapts" to your load, but it's not changing it's behavior in this case and, unlike other runtimes, there seems to be little I can do to tune it (have tried workstation GC, but there is very little observable difference).

user426445
  • 364
  • 3
  • 15
  • 1
    You best bet is to find out what objects you're allocating and make sure you only allocate what you need and don't hold on to instances longer than needed. Perfview is a good tool for digging into this. – Brian Rasmussen Jul 13 '15 at 18:06
  • G2 objects simply mean objects with long time span, as they are persisted by maintaining the reference, so are you releasing the objects as the need finish or it happens in the end. GC does adapts, in fact it generally gets invoked when there's a memory pressure, not when we think it should, it is undeterministic – Mrinal Kamboj Jul 13 '15 at 18:08
  • Hi, objects are released as soon as a request is over, typically living for a few seconds at most - but in that few seconds, we're seeing several GC collections happening and therefore (undesirable) promotion between the generations. – user426445 Jul 14 '15 at 07:34
  • @Brian - thanks for the suggestion, we'll give Perfview a whirl. – user426445 Jul 14 '15 at 07:37

2 Answers2

0

Microsoft decided to design the String class so that all strings are stored in memory as a monolithic sequence of characters. While this works well for some usage patterns, it works dreadfully for others.

One thing I've found very helpful is to avoid creating instances of String whenever possible. If a method will often be used to operate on part of a supplied string, and will in turn ask other methods to operate on parts of it, the methods should accept arguments specifying the range of the String upon which they should operate. This will avoid the need for callers of the first method to use Subst to construct a new String for the method to act upon, and will avoid the need to have the method call Subst to feed portions of the string to its callers. In some cases where I have used this technique, the creation of thousands of String instances--some quite large--could be replaced with zero.

supercat
  • 77,689
  • 9
  • 166
  • 211
  • Hi - based on what I've gathered so far, I believe that ultimately, this is what we need to do. We were hoping that we could GC tune our way out of this and keep our code simple, but suspect we're way off the GC sweet-spot with what we're doing, so will have to bend our code to work with, rather than against, the garbage collector. I'll see what we can do. It still feels odd that working with large strings can cause these sorts of problems. – user426445 Jul 14 '15 at 07:41
  • @user426445: Performance of certain `String` operations could be improved quite a bit if there existed methods to copy a *portion* of a string's content to an existing `StringBuilder` or `Char[]`. Otherwise code which needs to append half of a string to a `StringBuilder` either needs to copy the whole thing to a `StringBuilder` and then copy the appropriate portion from there, or else create a new `String` object containing the necessary range of material and then copy that. Both approaches require doing more than double the work that should be required. – supercat Jul 14 '15 at 15:34
  • Changes to string handling (using streams where possible, eliminating duplication/re-manipulation/etc) solved the problem... GC far better behaved now. – user426445 Jul 15 '15 at 07:46
0

CLR's GC "adapts" to your load

It can't know how much memory you are willing to tolerate as overhead. Here, you probably want to give the app like 5GB of heap so that collections are much rarer. The GC has no built-in tuning knobs for that (subjective note: that's a pitty).

You can force bigger heap sizes by using one of the low latency modes for short durations. That should cause the GC to try hard to avoid G2 collections. Monitor the RAM usage and disable low latency mode when consumption reaches 5GB.

This is a risky strategy but it's the best I think you can do.

I would not do it. You can maximally gain 2x throughput. Your CPU is maxed out, right? Workstation GC does not scale to multiple cores and leaves CPUs unused.

boot4life
  • 4,966
  • 7
  • 25
  • 47
  • Thanks - I'll look into the low latency mode stuff now - we're willing to try anything (one suggestion was for off-heap allocation - yikes!). – user426445 Jul 14 '15 at 07:43