1

I have a large (500K lines) .NET CF (C#) program, running on CE6/.NET CF 3.5 (v.3.5.10181.0). This is running on a FreeScale i.Mx31 (ARM) @ 400MHz. It has 128MB RAM, with ~80MB available to applications. My app is the only significant one running (this is a dedicated, embedded system). Managed memory in use (as reported by GC.Collect) is about 18MB. To give a better idea of the app size, here's some stats culled from .NET CF Remote Performance Monitor after staring up the application:

GC:
 Garbage Collections            131 
 Bytes Collected by GC          97,919,260
 Managed Bytes in use after GC  17,774,992
 Total Bytes in use after GC    24,117,424
 GC Compactions                 41
JIT:
 Native Bytes Jitted:           10,274,820
Loader:
 Classes Loaded                 7,393
 Methods Loaded                 27,691

Recently, I have been trying to track down a performance problem. I found that my benchmark after running the app in two different startup configurations would run in approximately 2 seconds (slow case) vs. 1 second (fast case). In the slow case, the time for the benchmark could change randomly from EXE run to EXE run from 1.1 to 2 seconds, but for any given EXE run, would not change for the life of the application. In other words, you could re-run the benchmark and the time for the test stays the same until you restart the EXE, at which point a new time is established and consistent.

I could not explain the 1.1 to 2x slowdown via any conventional mechanism, or by narrowing the slowdown to any particular part of the benchmark code. It appeared that the overall process was just running slower, almost like a thread was spinning and taking away some of "my" CPU.

Then, I randomly discovered that just by switching away from my app (the GUI loses the foreground) to another app, my performance issue disappears. It stays gone even after returning my app to the foreground. I now have a tentative workaround where my app after startup launches an auxiliary app with a 1x1 size window that kills itself after 5ms. Thus the aux app takes the foreground, then relinquishes it.

The question is, why does this speed up my application?

I know that code gets pitched when a .NET CF app loses the foreground. I also notice that when performing a "GC Heap" capture with .NET CF Remote Performance Monitor, a Code Pitch is logged -- and this also triggers the performance improvement in my app. So I suspect somehow that code pitching is related or even responsible for fixing performance. But I'm at a loss as to figure out how to determine if that is really the case, or even to explain why pitching code could help in this way. Does pitching out lots of code somehow significantly help locality of reference of code pages (that are re-JITted, presumably near each other in memory) enough to help this much? (My benchmark spans probably 3 dozen routines and hundreds of lines of code.)

Most importantly, what can I do in my app to reliably avoid this slower condition. Any pointers to relevant .NET CF / JIT / Code pitching information would be greatly appreciated.

Wil S
  • 345
  • 2
  • 12

3 Answers3

1

Your app going to the background auto-triggers a GC.Collect, which collects, may compact the GC Heap and may pitch code. Have you checked to see if a manual GC.Collect without going to the background gives the same behavior? It might not be pitching that's giving the perf gain, it might be collection or compaction. If a significant number of dead roots are swept up, walking the root tree may be getting faster. Can't say I've specifically seen this issue, so this is all conjecture.

ctacke
  • 66,480
  • 18
  • 94
  • 155
  • The app behavior is not affected by calls to System.GC.Collect. I make that call manually before the benchmark to ensure that I don't get a GC during the test which affects my timing. The behavior seems to persist through multiple GC cycles, including compaction. – Wil S Mar 09 '12 at 23:00
  • You've probably tried, but how about calling GC.Collect, snapshot, send to back, snapshot, bring to fore, snapshot. Is there a large change from after GC to going to the background (i.e. more objects collected, etc)? Just trying to pinpoint it to a compaction. – ctacke Mar 09 '12 at 23:13
  • Oh, and you can infer from the last comment that GC performance shouldn't be a factor in the benchmark, as GC does not run during the benchmark due to my prior running of GC. – Wil S Mar 09 '12 at 23:25
  • I take it by snapshot you mean a NETCF RPM GC Heap snapshot? I've not tried that, but I will do. It will take me a few minutes. – Wil S Mar 09 '12 at 23:26
  • OK, the first snapshot shows 90 GCs, 47 compactions, 0 pitches. It also shows managed byte in use after GC: 16,371,748. At the 2nd snapshot, it shows 91 GCs, 47 compactions, 1 pitch, and 16,371,312 managed bytes in use after GC. 3rd snapshot shows no change from 2nd in those counters. Were there other counters you would like to see. It would be easier if I could attach the files, but I'm not sure I can. – Wil S Mar 10 '12 at 00:04
  • Sure sounds like the only difference is the code pitch. I have absolutely *no* idea why that would make a difference. Smells like a bug to me. – ctacke Mar 10 '12 at 02:59
  • I agree. I usually try not to ask questions like this unless I'm pretty sure it's _not_ my problem... Saves much embarrassment. :-) – Wil S Mar 10 '12 at 22:50
1

Send your app a wm_hibernate before your performance loop. Will clean up things

pwf
  • 11
  • 1
  • WM_HIBERNATE (0x3FF) does seem to trigger code pitch, but oddly only if the message is sent (call of SendMessage is) from the GUI thread. So I have to Invoke over to the GUI thread before sending the message. Thanks for the tip. – Wil S Apr 04 '12 at 21:40
0

We have a similar issue with our .NET CF application.

Over time, our application progressively slows down, eventually to a halt with what I anticipate is due to high CPU load, or as @wil-s says, as if thread is spinning consuming CPU. The only assumption / conclusion I've made to so far is either we have a rogue thread in our code, or there's an under the cover issue in .NET CF, maybe with the JITter.

Closing the application and re-launching returns our application to normal expected performance.

I am yet to implement code change to test issuing WM_HIBERNATE or launch a dummy app which quits itself (as above) to force a code pitch, but am fairly sure this will resolve our issue based on the above comments. (so many thanks for that)

However, I'm subsequently interested to know whether a root cause was ever found to this specific issue?

Incidentally and seemingly off topic (but bear with me), we're using a Freescale i.MX28 processor and are experiencing unpredictable FlashDisk corruption. Seeing 2K blocks of 0xFFs (erased blocks) in random files located on NAND Flash.

I'm mentioning this as I now believe the CPU and FlashDisk corruption issues are linked, due to this article as well as this one: https://electronics.stackexchange.com/questions/26720/flash-memory-corruption-due-to-electricals

In the article, @jwygralak67 comments:

I recently worked through a flash corruption issue, on a WinCE system, as part of a development team. We would sporadically find 2K blocks of flash that were erased. (All bytes 0xFF) For about 6 months we tested everything from ESD, to dirty power to EMI and RFI interference, we bought brand new devices and tracked usage to make sure we weren't exceeding the erase cycle limit and buring out blocks, we went through our (application level) software with a fine toothed comb.

In the end it turned out to be an obscure bug in the very low level flash driver code, which only occurred under periods of heavy CPU load. The driver came from a 3rd party. We informed them of the issue we found, but I don't know if they ever released a patch.

Unfortunately, we're yet to make contact with him.

With all of this in mind, potentially if we work around the high CPU load, maybe the corruption will no longer be present. Another conjecture case! On that assumption however, this doesn't give a firm root cause for either situation, which I'm desperately seeking!

Any knowledge or insight, however small, would be very gratefully received.

@ctacke - we've spoken before regarding OpenNETCF via email, so I'm pleased to see your name!

Community
  • 1
  • 1
cyberenergy
  • 321
  • 3
  • 4
  • What filesystem are you using that is getting corrupted? We have had very reliable service from YAFFS. Beware the WM_HIBERNATE workaround discussed if porting to WEC2013 - it will randomly crash your .NETCF application. But it also may not be necessary on WEC2013/.NETCF 3.9 - I don't know yet. – Wil S Apr 24 '15 at 21:58
  • Hi @Wil S. We're using ExFAT. We are confident that the issue is specific to us using NAND Flash and bug in a low level driver. Files stored on SD are unaffected. Also many thanks for the comments on WEC2013 / .NET CF 3.9. – cyberenergy Apr 27 '15 at 08:00
  • Also do you know if an issue been raised (by yourself or another) with Microsoft regarding the performance issue? I'm tempted to raise it myself using my MSDN entitlement for their investigation. I wonder if you have any of the original sources for your test scenarios to support the case? Many thanks again. – cyberenergy Apr 27 '15 at 08:05
  • I submitted this to MS back in Apr 2012. I did not have a reduced test case. Ultimately their support engineer sent me an email with this text (edited to fit space but otherwise verbatim, see next comment): – Wil S May 04 '15 at 21:42
  • "Compact Framework ... was designed primarily to provide a RAD UI environment for Mobile applications. ... It will not handle high-performance requirements, like low-level interrupts or sub-second timers with the same degree of accuracy that a native-code equivalent will. It is not designed for the implementation of high-performance applications. That is the role of native code on Mobile devices. ... It’s best use is as a UI wrapper for high-performance native code. It works better in human-time than in computer-time." – Wil S May 04 '15 at 21:45
  • So, I took from this: (1) We are not interested in investigating or fixing your specific slowdown issue and (2) We are categorically not interested in NETCF performance issues, and (3) DO NOT USE NETCF in applications requiring high performance. "Human time" only! So... caveat emptor! – Wil S May 04 '15 at 21:50