1

I am using IIRF - an ISAPI rewrite filter for pretty URL's. I haven't been able to get much help from the developer on these issues. I'm hoping by making some sense of this dump, so I can find the problematic area in the code and rebuild it myself. I am not super familiar with C, but can get around. Do I need to build this with debug symbols to get anything useful out of the dump?

These stack overflow exceptions are happening on our live production web server, so I'm not able to use debug mode to step into this code. What's happening is my application pool processes (w3wp.exe) are receiving this error event:

A process serving application pool 'dotNET Pool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '6924'. The data field contains the error number.

Can someone help me make some sense of this data? It's a dump from the IIS Debug Diagnostic tool. How do I interpret it and find the source of the exception? It appears to be an exception inside the 3rd party PCRE regular expressions library.

WARNING - DebugDiag was not able to locate debug symbols for IsapiRewrite4.dll, so the information below may be incomplete.

In w3wp__PID__3760__Date__06_23_2009__ Time_01_29_55PM__916__Second_Chance_Exception_C00000FD.dmp the assembly instruction at IsapiRewrite4!pcre_exec+12f9 in C:\IIRF1.2.15R5\IsapiRewrite4.dll has caused a stack overflow exception (0xC00000FD) when trying to write to memory location 0x00fc2be8 on thread 5

Type of Analysis Performed   Crash Analysis 
Machine Name  WEB
Operating System   Windows Server 2003 Service Pack 2 
Number Of Processors   4 
Process ID   8056 
Process Image   c:\WINDOWS\system32\inetsrv\w3wp.exe 
System Up-Time   0 day(s) 09:26:25 
Process Up-Time   0 day(s) 02:17:00 

Thread 4 - System ID 6624
Entry point   w3tp!THREAD_MANAGER::ThreadManagerThread 
Create time   6/23/2009 11:12:56 AM 
Time spent in user mode   0 Days 0:0:40.906 
Time spent in kernel mode   0 Days 0:0:6.312 

Function                         Arg 1        Arg 2        Arg 3     Source 
IsapiRewrite4!pcre_exec+12f9     08166a27     01b6741f     081669b8    
IsapiRewrite4!pcre_exec+2779     08166a27     01b6746b     081669b8    
IsapiRewrite4!pcre_exec+1660     08166a26     01b6741f     081669b8    
IsapiRewrite4!pcre_exec+2779     08166a26     01b6746b     081669b8    
IsapiRewrite4!pcre_exec+1660     08166a25     01b6741f     081669b8    
IsapiRewrite4!pcre_exec+2779     08166a25     01b6746b     081669b8    
IsapiRewrite4!pcre_exec+1660     08166a24     01b6741f     081669b8    
IsapiRewrite4!pcre_exec+2779     08166a24     01b6746b     081669b8    
IsapiRewrite4!pcre_exec+1660     08166a23     01b6741f     081669b8    
IsapiRewrite4!pcre_exec+2779     08166a23     01b6746b     081669b8    
IsapiRewrite4!pcre_exec+1660     08166a22     01b6741f     081669b8    
[...snip...]

Update on this debug process

I believe that I have found the culprit, after tweaking the IIS Debug Diagnostic tools to provide more information, I have found URL's like the following. It appears to be a similar to a SQL injection attack, but I do not believe to be SQL Injection.

[original_url]/Forum+Result:+%E8%F1%EF%EE%EB%FC%E7%EE%E2%E0%ED+%ED%E8 %EA%ED%E5%E9%EC+%22Rifsadasy%22;%EF%E8%EA%F2%EE%EA%EE%E4+%E4%E5%F8%E8 %F4%F0%EE%E2%E0%ED;%E7%E0%F0%E5%E3%E8%F1%F2%F0%E8%F0%EE%E2%E0%EB%E8%F1 %FC+100%25+%28%E2%EA%EB%FE%F7%E5%ED+%F0%E5%E6%E8%EC+%F2%EE%EB%FC%EA%EE +%F0%E5%E3%E8%F1%F2%F0%E0%F6%E8%E8%29;

Has anyone seen this type of attack? Do they know what it is? I've tried to decode this using HEX, Base64 and some others, but I'm not coming up with anything except this ASCII text:

èñïîëüçîâàí+íèêíåéì+"Rifsadasy";ïèêòîêîä+äåøèôðîâàí;çàðåãèñòðèðîâàëèñü+100%+(âêëþ÷åí+ðåæèì+òîëüêî+ðåãèñòðàöèè);

Cheeso
  • 189,189
  • 101
  • 473
  • 713
MaseBase
  • 800
  • 3
  • 8
  • 31

6 Answers6

3

I think the stack overflow is not due to a bug in the rewriter. It is due to a bug in the patterns used in the configuration of the filter configuration.

It is actually easy to create a single regex that loops endlessly, and neither PCRE nor IIRF have a way to prevent that. It is also possible to create logic loops in the rewrite rules, so that you redirect or rewrite endlessly. Again, there's no way for the filter to stop you from shooting yourself in the foot. You have to take care. These risks exist when using any rewriter that depends on PCRE, or any modular regex engine.

The stack overflow is happening in pcre_exec, which is where the regular expression is being executed. It's a degenerate case but one that should be handled in the configuration. A prior rule should have filtered out such invalid cases. Here's a post about using a rewriting filter as a security barrier.

Test early and often. Some people think that because the filter rules are "just configuration", it is not in need of rigorous testing. In general, that's not a safe assumption. Treat the IIRF rules like any code module. You're just using a different language.

Cheeso
  • 189,189
  • 101
  • 473
  • 713
  • Cheeso! SO GLAD to see you chime in here... I have been needing your help. My understanding was the IterationLimit setting prevented this from becoming a problem. Setting the IterationLimit to 5 or 10 meant that one URL would stop being rewritten after 10 rewrites. In re-reading the desc of this setting, it seems that the iteration limit only applies to one pass through the rules. Once a URL enters the top of the ruleset again, the iteration limit is reset. Is this correct? Either way, now I know to begin dissecting my rules to find out where the loop is getting caught. – MaseBase Jun 30 '09 at 18:02
2

PCRE is calling two functions recursively repeatedly, until it runs out of stack space. You can either try to upgrade the Rewrite DLL to one that isn't buggy, or try to change your rewrite rules so that it doesn't hit the PCRE bug.

Ana Betts
  • 73,868
  • 16
  • 141
  • 209
  • This is not a bug in the rewriter. This is a bug in the pattern and configuration. the stack overflow is happening in pcre_exec, which is where the regular expression is being executed. It's a degenerate case but one that should be handled in config (eg, drop all URLs with % in them). – Cheeso Jun 30 '09 at 01:34
  • Any Regex parser that can be convinced to stack overflow on a certain input has a bug; though I agree that it might be easiest to fix this in the configuration – Ana Betts Jun 30 '09 at 02:07
  • Any general-purpose regex allows repeating null groups, which immediately allows the possibility of infinite loops. Conclusion: *all* regex engines are buggy! – Cheeso Jun 30 '09 at 02:32
  • So you're saying all those web Regex sites can easily be brought down by typing in a repeating null group into it – Ana Betts Jun 30 '09 at 04:49
  • NO. I'm saying, it's possible to build a regex that recurses on itself, regardless of the input. And, I'm also saying (or I should have said), it is easier to build a broken regex with more complicated inputs. ANY regex parser that is general enough supports recursion, and unless it has a failsafe check in the recursion logic, it will overflow. PCRE has a failsafe in it (sas the doc), but it apparently is not working in this case. Anyway I worked with Mase to change his regex and avoid the problem. – Cheeso Jul 12 '09 at 05:59
2

looks like you've got pcre_exec recursing and using up all of your stack space. I would check and see what regular expressions you are using.

jcopenha
  • 3,935
  • 1
  • 17
  • 15
1

Based on that stack trace at the very bottom, it looks like the program is going into infinite recursion, with two functions that are calling each other without terminating. This will cause a stack overflow exception due to eventually running out of available stack.

Tyler McHenry
  • 74,820
  • 18
  • 121
  • 166
1

Can you figure out the URL that's sending into the infinite recursion? It looks like you have two rewrite rules that are triggering each other. Unless you have the source to IsapiRewrite4.dll, it's not going to help much -- you can get to the assembly code -- but even if you had the source (you could decompile), it won't help, because I think you need to see the URL that got passed.

Did it also dump memory (or could you make it do that). Arg1, Arg2, or Arg3 might be a pointer to the URL?

Lou Franco
  • 87,846
  • 14
  • 132
  • 192
  • I DO have the source for IsapieRewrite4.dll. It's open source, available on the [IIRF website](http://www.codeplex.com/IIRF) – MaseBase Jun 24 '09 at 20:06
  • 1. make a build with a pdb 2. deploy the dll with the pdb in the same dir 3. repro problem 4. (maybe debug diag will read pdb), if not, load IIS with the dll on your local machine and the pdb in the same directory. Find the function in the error. 5. add logging to the function so you know what URL is being rewritten -- see if you can figure out how many times the function is on the stack (thread local variable counter?), and if it's there more than a few times, log what is happening. 5. Run the new dll in prodcution. – Lou Franco Jun 25 '09 at 02:29
  • Thanks Lou... if I could repro the problem, I wouldn't be posting this question. Also, I tried deploying the DLL with the PDB, but the debug diag did not read the symbols, it's saying it can't find them. – MaseBase Jun 25 '09 at 20:47
  • ok, but now that you have a PDB, load IIS on your Dev box with the ISAPI DLL and go to the address in the debug diag -- you should be able to see symbols (put the PDB in the same directory as the DLL). I don't think this will help much as I think the problem is in the URL that was actually passed. Maybe you can patch by having the function detect that it's in a mutually recursive loop and exiting with an error or exception. – Lou Franco Jun 25 '09 at 20:55
  • You don't need a debuggable version of IIRF. It's clear the stack overflow is happening in the pcre_exec function, which is where the Regular Expression is evaluated/executed. It is overflowing because the rule is looping. This is not uncommon in regex processing. Be careful! – Cheeso Jun 30 '09 at 01:43
1

So I believe I have found the cause error. Though I'm not sure if this is still really a bug in IIRF ISAPI filter? It at least should not iterate through the rewrite functions so many times that it causes a stack overflow.

I can reproduce the error I was seeing in the event log by requesting this URL on my server: [original_url]/Forum+Result:+%E8%F1%EF%EE%EB%FC%E7%EE%E2%E0%ED+%ED%E8 %EA%ED%E5%E9%EC+%22Rifsadasy%22;%EF%E8%EA%F2%EE%EA%EE%E4+%E4%E5%F8%E8 %F4%F0%EE%E2%E0%ED;%E7%E0%F0%E5%E3%E8%F1%F2%F0%E8%F0%EE%E2%E0%EB%E8%F1 %FC+100%25+%28%E2%EA%EB%FE%F7%E5%ED+%F0%E5%E6%E8%EC+%F2%EE%EB%FC%EA%EE +%F0%E5%E3%E8%F1%F2%F0%E0%F6%E8%E8%29;

So I've created a rewrite rule to return a 404 status for this URL.

But I needed to know if this was some type of attack, I cannot be fully sure yet, but here is what I think the encrypted string says. The URL was coming from IP addresses in Russia, so here is what I've done to translate it:

  1. Hex -to- ASCII:

    èñïîëüçîâàí+íèêíåéì+"Rifsadasy";ïèêòîêîä+äåøèôðîâàí;çàðåãèñòðèðîâàëèñü+100%+(âêëþ÷åí+ðåæèì+òîëüêî+ðåãèñòðàöèè)

  2. Then Cyrillic to Russian:

    использован+никнейм+"Rifsadasy";пиктокод+дешифрован;зарегистрировались+100%+(включен+режим+только+регистрации)

  3. Then Russian to English:

    used nickname "Rifsadasy"; piktokod decrypt; registered 100 (mode only)
    or, similarly
    It is used никнейм "Rifsadasy"; пиктокод it is decoded; were registered 100 (the mode only registration is included)

MaseBase
  • 800
  • 3
  • 8
  • 31