What problems would arise in constructing a Java Decompiler through a grammar?

Question

I am currently constructing a Java decompiler.

In order to assist in pattern recognition, I am constructing a simple grammar through ANTLR and using ANTLRWorks interpreter to debug.

Below's the preliminary grammar so far. In going down this route, I am assuming that I am able to simplify certain JVM byte-code into expressions that the grammar below can detect.

What problems do you see in this approach? Updated grammar for Ira's comments on June 29 2:36 GMT 1

    grammar JVM;

options {k=3;}

WS  :   (' '|'\r'|'\n'|'\t')+ {$channel=HIDDEN;}
    ;
INT :   ('0'..'9')+ ;
UINT    :   ('_' INT)?;
IFEQ    :   'ifeq';
IFGE    :   'ifge';
IFGT    :   'ifgt';
IFLE    :   'ifle';
IFLT    :   'iflt';
IFNE    :   'ifne';
IFACMP_CONDTYPE :   'if_acmp' ('eq'|'ne'|'lt'|'ge'|'gt'|'le');
// THIS :   'aload_0';
LDC :   'ldc2_w'|'ldc_w'|'ldc';
LOADREFERENCE
//  :   THIS
    : 'aload' UINT;
//  | 'aload_2'
//  | 'aload_3';
DLOAD   :   'dload' UINT;
LOADINT :   'iload_0'
    |   'iload_1'
    | 'iload_2'
    | 'iload_3'
    ;
DCONST  :   'dconst' UINT;  
ICONST  :   'iconst' UINT;

goal    :   jvmStatement2+ ;

//fragment
//jvmStatement1
//  :   returnStatement
//  | newArrayStatement
//  | storeStatement
//  | assignmentStatement
//  | assertStatement
//  | invokeStatement
//  | ifStatement
//  | gotoStatement
//  ;

fragment // to test assert
jvmStatement2
    : returnStatement     // 2
    | newArrayStatement   // 3
    | storeStatement      // 4
    | invokeStatement     // 5
    | assignmentStatement // 6
    | assertStatement     // 7
    | ifStatement         // 8  
    | gotoStatement
    ;

fragment
setAssertionStatus
    :   ifStatement pushIntegerConstant
    gotoStatement pushIntegerConstant setStaticFieldInClass;

fragment
fetchFieldFromObject
    :   LOADREFERENCE 'getfield' INT;

fragment
loadDoubleFromLocalVariable
    :   DLOAD;

fragment
loadFloatFromLocalVariable
    :   'fload' UINT;

fragment
loadIntFromLocalVariable
    :   LOADINT;

fragment
loadLongFromLocalVariable
    :   'lload' UINT;   

fragment
loadReferenceFromLocalVariable
    :   'aload' UINT;

fragment
loadReferenceFromArray
    :   'aaload';

fragment
storeReference
    : storeIntoByteOrBooleanArray;  

fragment
storeReferenceIntoLocalVariable
    :   'astore' UINT;

fragment
storeDoubleIntoLocalVariable
    :   'dstore' INT;

fragment
storeFloatIntoLocalVariable
    :   'fstore' UINT;

fragment
storeIntIntoLocalVariable
    :   'istore' (INT|UINT);

fragment
storeLongIntoLocalVariable
    :   'lstore' UINT;  

fragment
storeIntoByteOrBooleanArray
    :   'bastore';

fragment
storeIntoReferenceArray
    :   'aastore';

fragment
pushNull:   'aconst_null';

fragment
pushByte:   'bipush' INT;

fragment
pushIntegerConstant
    :   ICONST;

fragment
pushDoubleConstant
    :   DCONST;

fragment
pushLongConstant
    :   'lconst' UINT;

fragment
pushFloatConstant
    :   'fconst' UINT;

fragment
pushItemFromRuntimeConstantPool
    :   LDC INT;


fragment invokeStatementArgument: constantExpr
    | createAnonymousClass;

fragment createAnonymousClass
    :   createNewObject dup thisInstance;

fragment invokeStatementArguments: invokeStatementArgument*;

fragment invokeStatement: getStaticField? invokeStatementArguments invokeMethod;    

fragment
invokeMethod
    : invokeInstanceMethod
    | invokeVirtualMethod
    | invokeStaticMethod
    ;

fragment
invokeInstanceMethod
    :   'invokespecial' INT;

fragment
invokeVirtualMethod
    :   'invokevirtual' INT;    

fragment
invokeStaticMethod
    :   'invokestatic' INT;

fragment
newArrayStatement
    :   'newarray' simpleType;

fragment
setFieldInObject
    :   'putfield' INT;

fragment setStaticFieldInClass
    :   'putstatic' INT;

fragment
simpleType
    :   ('boolean'|'byte'|'char'|'double'|'float'|'int'|'long'|'short');

fragment
returnVoid
    :    'return';
fragment
returnSimpleType
    :   returnReference
    | returnDouble
    | returnFloat
    | returnInteger
    | returnLong;

fragment
returnReference
    :    'areturn';
fragment
returnDouble
    :   'dreturn';
fragment returnFloat
    :   'freturn';
fragment returnInteger
    :   'ireturn';
fragment returnLong
    :   'lreturn';  

fragment
returnStatement
    :   returnVoid 
    | constantExpr returnSimpleType;    

fragment
dupX1
    :   'dup_x1';

fragment
dup
    :   'dup';  

fragment
storeStatement
    : storeReferenceIntoLocalVariable 
    | storeIntIntoLocalVariable
    | setStaticFieldInClass
    | storeIntoReferenceArray
    | setFieldInObject;

fragment
convertDouble
    :   convertDoubleToFloat | convertDoubleToInt | convertDoubleToLong;

fragment
convertDoubleToFloat
    :   'd2f';

fragment
convertDoubleToInt
    :   'd2i';

fragment
convertDoubleToLong
    :   'd2l';

fragment
convertFloat
    :   convertFloatToDouble|convertFloatToInt|convertFloatToLong;

fragment
convertFloatToDouble
    :   'f2d';
fragment
convertFloatToInt
    :   'f2i';
fragment
convertFloatToLong
    :   'f2l';  

fragment
convertInt
    :   convertIntToByte
    |convertIntToChar
    |convertIntToDouble
    |convertIntToFloat
    |convertIntToLong
    |convertIntToShort;

fragment
convertIntToByte
    :   'i2b';

fragment
convertIntToChar
    :   'i2c';

fragment
convertIntToDouble
    :   'i2d';

fragment
convertIntToFloat
    :   'i2f';

fragment
convertIntToLong
    :   'i2l';

fragment
convertIntToShort
    :   'i2s';

fragment
branchComparison
    :branchIfReferenceComparison
    |branchIfIntComparison
    |branchIfIntComparisonWithZero
    |branchIfReferenceNotNull
    |branchIfReferenceNull; 

fragment
branchIfReferenceComparison
    :   'if_acmp' condType;

fragment
branchIfIntComparison
    :   'if_icmp' condType INT;

fragment
branchIfIntComparisonWithZero
    :   (IFEQ|IFGE|IFGT|IFLE|IFLT|IFNE) INT;

fragment
gotoStatement
    :   'goto' INT;

fragment
ifStatementCompare
    :   (IFEQ INT)
    |   (IFNE INT);

fragment
ifStatement
    :   booleanExpression ifStatementCompare;

fragment
ifType  : 'ifeq'
 |'ifne'
 |'iflt'
 |'ifge'
 |'ifgt'
 |'ifle';

fragment
branchIfReferenceNotNull
    :   'ifnonnull' ;

fragment
branchIfReferenceNull
    :   'ifnull';

fragment
condType:   'eq'
 |'ne'
 |'lt'
 |'ge'
 |'gt'
 |'le';

fragment
checkCast
    :   'checkcast' INT;

fragment
createNewArrayOfReference
    :   constantExpr 'anewarray' INT;

fragment
createNewObject
    :   'new' INT;

fragment
assignmentStatement
//  : pushItemFromRuntimeConstantPool storeStatement
    : (constantExpr)+ storeStatement
    | invokeInheritedConstructor
    | expressionStatement
//  | setAssertionStatus
    ;

fragment
invokeInheritedConstructor
    :   loadReferenceFromLocalVariable invokeInstanceMethod;

fragment
throwExceptionOrError
    :   'athrow';

fragment
getStaticField
    :   'getstatic' INT;

fragment
newInstance
    :   'new' INT;

fragment // this needs to be extended to recognize more patterns
booleanExpression
    :   integerComparison
    | loadIntFromLocalVariable
    | invokeMethod;

fragment
integerComparison
    : loadIntFromLocalVariable loadIntFromLocalVariable branchIfIntComparison;  

fragment assertIfAssertEnabled: getStaticField branchIfIntComparisonWithZero;

fragment assertCondition:booleanExpression branchIfIntComparisonWithZero;

fragment assertThrow:createNewObject dup assertMessage throwExceptionOrError;

fragment assertMessage:pushItemFromRuntimeConstantPool invokeMethod;

fragment assertStatement:assertIfAssertEnabled assertCondition assertThrow;


fragment
stringPlusNumber
    :pushItemFromRuntimeConstantPool invokeMethod 
 loadReferenceFromLocalVariable invokeMethod invokeMethod invokeMethod;

fragment expressionStatement:   statementExpression;

fragment
statementExpression 
    :   preIncrementExpression
    | preDecrementExpression
//  | postIncrementExpression
//  | postDecrementExpression
    | newByteArray
    | ternaryExpression
    | createAndStoreObject // assignment expression
    | createNewArrayStatement
    | fetchFieldFromObject
    ;

fragment
createNewArrayStatement // with elements
    :   createNewArrayOfReference createNewArrayInitElement+;

createNewArrayInitElement
    : (dup constantExpr getStaticField storeStatement);

fragment
createAndStoreObject
    :   createNewObject dup invokeStatement storeStatement;

fragment ternaryExpression // doesn't cover all situations yet
    : loadIntFromLocalVariable ifStatementCompare loadIntFromLocalVariable gotoStatement
    loadIntFromLocalVariable storeStatement;    

fragment preIncrementExpression: preIncrementInteger;

fragment preDecrementExpression: preDecrementFloat|preDecrementLong|preDecrementDouble; 

fragment doubleExpression: pushDoubleConstant;

fragment integerExpression: pushIntegerConstant;

fragment longExpression: pushLongConstant;

fragment floatExpression: pushFloatConstant;

fragment preIncrementInteger: loadReferenceFromLocalVariable dup fetchFieldFromObject integerExpression 
    iAdd dupX1? setFieldInObject;

fragment preDecrementDouble: loadDoubleFromLocalVariable doubleExpression dSub storeDoubleIntoLocalVariable;

fragment preDecrementLong: loadLongFromLocalVariable longExpression lSub storeLongIntoLocalVariable;

fragment preDecrementFloat: loadFloatFromLocalVariable floatExpression fSub storeFloatIntoLocalVariable;

fragment newByteArray: newByteArrayWithNull|newByteArrayWithData;

// byte[] b = {'c', 'h', 'u', 'a'};
fragment newByteArrayWithData:  constantExpr newArrayStatement byteArrayElements;

fragment byteArrayElements: constantExpr constantExpr storeIntoByteOrBooleanArray;  

fragment constantExpr: 
    //loadReferenceFromLocalVariable
    LOADREFERENCE
    |loadDoubleFromLocalVariable
    |loadFloatFromLocalVariable
    |loadIntFromLocalVariable
    |loadLongFromLocalVariable
    |pushByte
    |pushDoubleConstant
    |pushFloatConstant
    |pushIntegerConstant
    |pushItemFromRuntimeConstantPool
    |pushLongConstant
    |pushNull
    |fetchFieldFromObject
    ;

// byte[] c = null;
// String s = null;
fragment newByteArrayWithNull: pushNull (checkCast)? storeReference;

fragment thisInstance:  LOADREFERENCE invokeMethod;

fragment ternaryOperator
    :   ifStatementCompare pushIntegerConstant gotoStatement pushIntegerConstant setStaticFieldInClass;

fragment floatMultiply
    :   constantExpr constantExpr dMul;

fragment iAdd: 'iadd';      
fragment dSub: 'dsub';
fragment fSub: 'fsub';
fragment lSub: 'lsub';
fragment lAdd: 'ladd';  
fragment dMul: 'dmul';

For example, the current grammar (a further evolution of the above) can turn

getstatic 25
ifne 25
iload_1
iload_2
if_icmpgt 25
new 25
dup
invokespecial 44
athrow
return

into

Answer 1

If all you want to recognize are the individual JVM instructions then a grammar might be OK. You'll probably spend time fiddling with the grammar to get the details right. This might be simple overkill. A byte-opcode-driven finite state automaton (FSA) implemented as a giant case statement might be easier; after all, the JVM instructions are supposed to be easy to decode so that a semi-fast interpreter could execute those instructions.

Based on vague recall, there are other sections (tables, e.g., literals) in the class file. You can probably recognize them, too, with the parser but also likely overkill.

You have the second problem of collecting the instruction/table information after you recognize them; parser generators tend to want to help you build some kind of AST. The instructions aren't an AST; they're at least a linear chain and if you include the jump targets, they form a graph with references to the tables. So I suspect you'll end up struggling to get the semantic actions to collect the data the way you want.

Ands its the graph you likely want to capture. To the extent that the graph has some kind of hierarchical structure (being derived from a structured programming language), you might want to discover that hierarchy. The parser approach contributes nothing here.

Answer 2

This approach has the problem of recognizing the nesting of parameters.

For example, given the declaration,

int func1(int x, int y, int z) {
    return 0;
}

int func0() {
    return 0;
}

and the call

Object[] x = new Object[func1(2, 3, 4)];
x = new Object[func0()];
x = new Object[func1(func1(func1(0, 1, 2), 3, 4), 5, 6)];

which generates the following bytecode:

Offset  Instruction       Comments (Method: none)
0       aload_0           (cheewee.helloworld.test000031_applet this)
1       iconst_2
2       iconst_3
3       iconst_4
4       invokevirtual 79  (cheewee.helloworld.test000031_applet.func1)
7       anewarray 81      (java.lang.Object)
10      astore_1          (java.lang.Object[] x)
11      aload_0           (cheewee.helloworld.test000031_applet this)
12      invokevirtual 83  (cheewee.helloworld.test000031_applet.func0)
15      anewarray 81      (java.lang.Object)
18      astore_1          (java.lang.Object[] x)
19      aload_0           (cheewee.helloworld.test000031_applet this)
20      aload_0           (cheewee.helloworld.test000031_applet this)
21      aload_0           (cheewee.helloworld.test000031_applet this)
22      iconst_0
23      iconst_1
24      iconst_2
25      invokevirtual 79  (cheewee.helloworld.test000031_applet.func1)
28      iconst_3
29      iconst_4
30      invokevirtual 79  (cheewee.helloworld.test000031_applet.func1)
33      iconst_5
34      bipush 6
36      invokevirtual 79  (cheewee.helloworld.test000031_applet.func1)
39      anewarray 81      (java.lang.Object)
42      astore_1          (java.lang.Object[] x)
43      return

It is unable to detect that there are nesting involved. I'm not sure if this is a limitation of ANTLR or if this is a limitation of my learnings on how to write an ANTLR grammar.

The next step would be to use a hybrid method, simplifying groups of bytecode into tokens (so as to recognize them as simpler patterns) first, before passing it into a parser for detecting higher level patterns.