Disassembling file that contain big data or is compressed

Question

this is fourth day I am trying to figure out how to break down an exe.

Still having no luck, file is giving debugger error right after it is runned. I am using OllyDBG, it seems that file is either compressed or contains big ammount of data. I think it is just for debugging protection, however I can not get it working.

I am trying to learn assembly and this is my "new level" achievment of getting better in testing applications. All I want to change is one text to other, inside the file exe. So this is one variable change. I would be satisfied even with simple number change inside it. Just want to know how.

The file orginally runs other exe after it is opened, but this is not anything I want to touch or edit.

This is how the file open:

00401000 >/$ 68 01504400    PUSH tryme.00445001
00401005  |. E8 01000000    CALL tryme.0040100B
0040100A  \. C3             RETN
0040100B   $ C3             RETN
0040100C     A9             DB A9
0040100D     FE             DB FE
0040100E     39             DB 39                                    ;  CHAR '9'
0040100F     B1             DB B1
00401010     30             DB 30                                    ;  CHAR '0'
00401011     D8             DB D8
00401012     BB             DB BB
00401013     A6             DB A6
00401014     45             DB 45                                    ;  CHAR 'E'
00401015     23             DB 23                                    ;  CHAR '#'
00401016     92             DB 92
00401017     AC             DB AC
00401018     3D             DB 3D                                    ;  CHAR '='
00401019     B3             DB B3
0040101A     9C             DB 9C
0040101B     8C             DB 8C
0040101C     90             NOP
0040101D     0E             DB 0E
0040101E     26             DB 26                                    ;  CHAR '&'
0040101F     3B             DB 3B                                    ;  CHAR ';'
00401020     D3             DB D3
00401021     48             DB 48                                    ;  CHAR 'H'
00401022     49             DB 49                                    ;  CHAR 'I'
00401023     70             DB 70                                    ;  CHAR 'p'
00401024     88             DB 88
00401025     07             DB 07
00401026     78             DB 78                                    ;  CHAR 'x'
00401027     36             DB 36                                    ;  CHAR '6'
00401028     7C             DB 7C                                    ;  CHAR '|'
00401029     88             DB 88

below this there are many DB calls, I tried to breakpoint every other RETN, but they are not called. Can someone give me a hint, how to deal with this kind of exe files?

Thank you for your time,

Answer 1

udis86 disassembler library has a very useful and handy disassembler called udcli.

For example, what I did to understand your code:

First, copy all the hex code bytes into an ASCII file. I copied your OllyDbg output and then cut off with Vim everything except the binary code, resulting in a text file like this (let's say hexcode.txt):

68 01 50 44 00 E8 01 00 00 00 C3 C3 A9 FE 39 B1 30 D8 BB A6 45 23 92 AC 3D B3 9C 8C 90 0E 26 3B D3 48 49 70 88 07 78 36 7C 88

Then wondering whether this is 16-bit, 32-bit or 64-bit Intel code... usually you can see and feel if the code seems strange, in that case it's either wrong processor, wrong processor mode or the code may be encrypted or it may be data and not code.

Let's try if it's 16-bit code:

In Linux console, $ cat hexcode.txt | udcli -x -16

0000000000000000 680150           push word 0x5001        
0000000000000003 44               inc sp                  
0000000000000004 00e8             add al, ch              
0000000000000006 0100             add [bx+si], ax         
0000000000000008 0000             add [bx+si], al         
000000000000000a c3               ret                     
000000000000000b c3               ret                     
000000000000000c a9fe39           test ax, 0x39fe         
000000000000000f b130             mov cl, 0x30            
0000000000000011 d8bba645         fdivr dword [bp+di+0x45a6]
0000000000000015 2392ac3d         and dx, [bp+si+0x3dac]  
0000000000000019 b39c             mov bl, 0x9c            
000000000000001b 8c900e26         mov [bx+si+0x260e], ss  
000000000000001f 3bd3             cmp dx, bx              
0000000000000021 48               dec ax                  
0000000000000022 49               dec cx                  
0000000000000023 7088             jo 0xffffffffffffffad   
0000000000000025 07               pop es                  
0000000000000026 7836             js 0x5e                 
0000000000000028 7c88             jl 0xffffffffffffffb2   

Hmmm. Already in the beginning inc sp, very strange instruction. Conclusion: not 16-bit code.

Maybe it's 32-bit code?

$ cat hexcode.txt | udcli -x -32

0000000000000000 6801504400       push dword 0x445001     
0000000000000005 e801000000       call dword 0xb          
000000000000000a c3               ret                     
000000000000000b c3               ret                     
000000000000000c a9fe39b130       test eax, 0x30b139fe    
0000000000000011 d8bba6452392     fdivr dword [ebx+0x922345a6]
0000000000000017 ac               lodsb                   
0000000000000018 3db39c8c90       cmp eax, 0x908c9cb3     
000000000000001d 0e               push cs                 
000000000000001e 263bd3           cmp edx, ebx            
0000000000000021 48               dec eax                 
0000000000000022 49               dec ecx                 
0000000000000023 7088             jo 0xffffffffffffffad   
0000000000000025 07               pop es                  
0000000000000026 7836             js 0x5e                 
0000000000000028 7c88             jl 0xffffffffffffffb2   

This looks already better. First, you could set a breakpoint into 0x445001. As that dword gets pushed immediately before a call dword 0xb followed by a ret, it may be that the ret after call 0xb actually pops the value 0x445001 from stack and jumps to cs:0x445001. On the other hand, if there's an intent to obfuscate the code, it may be that the function called with call dword 0xb may modify the value 0x445001 pushed into stack, so that ret after call dword 0xb would not jump to 0x445001, but somewhere else. So set another breakpoint to the stack address where 0x445001 is stored. Before the function call call dword 0xb [ss:esp] should point to the value 0x445001, so set the breakpoint there. It can be set also inside the function, but in that case the address will be [ss:esp+4] ([ss:esp] holds the return address). So I would try first set these 2 breakpoints and then trace the code with single-stepping inside the call dword 0xb function.

A final thought: what if this is 64-bit code?

$ cat hexcode.txt | udcli -x -64

0000000000000000 6801504400       push dword 0x445001     
0000000000000005 e801000000       call dword 0xb          
000000000000000a c3               ret                     
000000000000000b c3               ret                     
000000000000000c a9fe39b130       test eax, 0x30b139fe    
0000000000000011 d8bba6452392     fdivr dword [rbx-0x6ddcba5a]
0000000000000017 ac               lodsb                   
0000000000000018 3db39c8c90       cmp eax, 0x908c9cb3     
000000000000001d 0e               invalid                 
000000000000001e 263bd3           cmp edx, ebx            
0000000000000021 48497088         jo 0xffffffffffffffad   
0000000000000025 07               invalid                 
0000000000000026 7836             js 0x5e                 
0000000000000028 7c88             jl 0xffffffffffffffb2

Begins the same way as 32-bit code, but later there's an invalid instruction, so probably it's not 64-bit code (unless the code hooks invalid opcode exception handler int 6), or never executes that code.