Extract the results of a GWT service

Question

Messy, complicated question, but here goes. I'm working on an integration project with Google Checkout, and there is a Google Checkout GWT service that returns the currency conversion rates used by the Checkout web interface to convert USD into local currencies. This endpoint is hosted at https://market.android.com/publish/gwt/, and staring at Firebug I see this going to the server:

7|0|6|https://market.android.com/publish/gwt/|FCCA4108CB89BFC2FEC78BA7363D4AF6|com.google.wireless.android.vending.developer.

shared.MerchantService|getCurrencyExchangeRates|com.google.common.money.CurrencyCode/112449834|java.util.ArrayList/4159755760

|1|2|3|4|2|5|6|5|235|6|13|5|18|5|81|5|53|5|72|5|102|5|121|5|177|5|175|5|205|5|204|5|55|5|86|-1|

and this being returned

//OK[235,3,'D0JA',2,86,3,'CXXg',2,55,3,'DW2A',2,204,3,'X9NA',2,205,3,'EuvA',2,175,3,'VIig',2,177,3,'E2Dw',2,121,3,'E4ziA',2,1

02,3,'do$Q',2,72,3,'T82w',2,53,3,'Ds0Q',2,81,3,'Cq5g',2,18,3,'Dlfg',2,13,1,["com.google.common.collect.RegularImmutableList/4

40499227","com.google.common.money.SimpleMoney/627983206","com.google.common.money.CurrencyCode/112449834"],0,7]

Forgive the odd formatting: can't quite get the code block to format right.

Wandering the web for hours on end I was able to determine that the RegularImmutableList class is in the Guava libraries (at http://code.google.com/p/guava-libraries/). What I'm looking for is:

1. I can't find the com.google.common.money.SimpleMoney or com.google.common.money.CurrencyCode classes anywhere: anyone seen them?
2. The GWT wire format appears to be an odd JSON string. I see various references to Google Groups messages talking about descriptions of the wire format, but can't find the underlying messages or any coherent reference that would let me reverse this: anyone have a handle on a handy reference? If I can at least understand WHAT the encoding is I might be able to get away without the class files from question 1 above.
3. I started wandering through the Android Market api library at http://code.google.com/p/android-market-api/, figuring they have to have done SOME of the Android Market communication integration, and they appear to have done so using protobufs. Is there any decent reference for the GWT/protobufs communication bits?

The underlying reason for this craziness is that I need to be able to take regular exchange rate values from Google Checkout so when I'm importing sales transactions in foreign currencies I can do the conversion at the prevailing rate at the time of the transaction. The current Checkout reporting formats do NOT provide this, so most folks end up using alternative sources of exchange rates that don't match what Google uses. It is clearly a shortcoming on the part of Google Checkout's integration interface, but if we got started on shortcomings of Google Checkout's interface we'd be here all week. My intention is to poll the Checkout interface for newly fulfilled orders and then request the appropriate exchange rate table so I can figure out in near real-time what the incoming payments are. I've got the polling bit down pat but can't quite get past the exchange rate bit.

Answer 1

While trying to create a script to bulk upload in-app products for my application (CSV upload constantly failed with obscure error messages), I have managed to understand the GWT AJAX protocol.

It's actually pretty simple, except it requires you to know structure of all used classes. Or guess it, as is the case with internal classes used by Google. :)

I'll use examples from the question to explain the protocol in detail.

---

Request format

7|0|6|https://market.android.com/publish/gwt/|FCCA4108CB89BFC2FEC78BA7363D4AF6|com.google.wireless.android.vending.developer.shared.MerchantService|getCurrencyExchangeRates|com.google.common.money.CurrencyCode/112449834|java.util.ArrayList/4159755760|1|2|3|4|2|5|6|5|235|6|13|5|18|5|81|5|53|5|72|5|102|5|121|5|177|5|175|5|205|5|204|5|55|5|86|-1|

The request is pipe-delimited list of tokens with the following meaning:

1. 7 - protocol version
2. 0 - flags. 1 is FLAG_ELIDE_TYPE_NAMES, 2 is FLAG_RPC_TOKEN_INCLUDED
3. 6 - string token count
4. 6 string tokens:
   1. https://market.android.com/publish/gwt/
   2. FCCA4108CB89BFC2FEC78BA7363D4AF6
   3. com.google.wireless.android.vending.developer.shared.MerchantService
   4. getCurrencyExchangeRates
   5. com.google.common.money.CurrencyCode/112449834
   6. java.util.ArrayList/4159755760
5. The actual encoded request, which references strings from the list above using 1-based indices:
   1. 1 - https://market.android.com/publish/gwt/ - base URL
   2. 2 - FCCA4108CB89BFC2FEC78BA7363D4AF6 - some hash, which is references as serializationPolicyStrongName in GWT sources.
   3. 3 - com.google.wireless.android.vending.developer.shared.MerchantService - service name
   4. 4 - getCurrencyExchangeRates - method name
   5. 2 - parameter count. Parameter types follow:
      1. 5 - com.google.common.money.CurrencyCode/112449834
      2. 6 - java.util.ArrayList/4159755760
   6. Serialized parameters. Each object is represented either by its classname and list of serialized fields or by negative integer back-reference to previously encountered object. In our case we have two objects:
      1. 5 - com.google.common.money.CurrencyCode/112449834, which only has one integer field: 235
      2. 6 - java.util.ArrayList/4159755760, which has one integer length field 13, followed by 13 serialized list items. Note that 12 of them are CurrencyCode objects serialized just as the above one, and the last one is a backreference (-1) to the very first object we encountered while (de-)serializing this request, i.e. CurrencyCode(235)

---

Response format

//OK[235,3,'D0JA',2,86,3,'CXXg',2,55,3,'DW2A',2,204,3,'X9NA',2,205,3,'EuvA',2,175,3,'VIig',2,177,3,'E2Dw',2,121,3,'E4ziA',2,102,3,'do$Q',2,72,3,'T82w',2,53,3,'Ds0Q',2,81,3,'Cq5g',2,18,3,'Dlfg',2,13,1,["com.google.common.collect.RegularImmutableList/440499227","com.google.common.money.SimpleMoney/627983206","com.google.common.money.CurrencyCode/112449834"],0,7]

The response is very similar in format to the request except it's JS-formatted array (though not JSON, as it uses invalid single quotes), and it's in reverse order. The field meaning is as follows:

1. 7 - protocol version
2. 0 - flags, same as for request
3. Array of string tokens:
   1. com.google.common.collect.RegularImmutableList/440499227
   2. com.google.common.money.SimpleMoney/627983206
   3. com.google.common.money.CurrencyCode/112449834
4. And then goes one serialized object of type 1 - com.google.common.collect.RegularImmutableList/440499227 with one integer length field 13, followed by 13 serialized objects of class 2 - com.google.common.money.SimpleMoney/627983206. Each SimpleMoney object has two fields, for example:
   1. 'Dlfg' - long integer field encoded as base64 number. This particular one is 940000
   2. 3, 18 - CurrencyCode object with integer field 18

Answer 2

I have some code made in VB that may be useful for you to realize how to parse GWT Serialized strings. "Datos" contains the string you received.

aAux = Split(Datos, ",[")
aAux(1) = Replace(aAux(1), "],0,7]", "")
aAux(0) = Replace(aAux(0), "//OK[", "")
aAux(0) = Replace(aAux(0), "'", "")

aDescripcion = Split(aAux(1), """,""")
aValor = Split(aAux(0), ",")
InvertirArray aValor

For X = 0 To UBound(aValor)
    If Not IsNumeric(aValor(X)) Then
        Exit For
    End If
    If adescripcion(Int(aValor(X))-1) = "gov.senasa.embalajemadera.shared.domain.Pais/3238585366" Then  
        For Y = X + 1 To UBound(aValor)
            If Int(aValor(Y)) = "" Then '- Do what you want 
            end if
            If adescripcion(Int(aValor(Y))) = "java.lang.Integer/3438268394" Then 
        '- Do what you want
        Next Y
    End If
Next X

Of course you have to adapt it to your needs and you will have to play a little bit with the arrays...

InvertirArray:

Public Sub InvertirArray(ByRef Arr() As String)
'- el array va tiene que empezar en 0
Dim X As Long
Dim Hasta As Long
Dim Tmp As String

If UBound(Arr) Mod 2 = 0 Then
   '- Es impar
    Hasta = UBound(Arr) + 1
Else
    Hasta = UBound(Arr)
End If

For X = LBound(Arr) To UBound(Arr) \ 2
    Tmp = Arr(X)
    Arr(X) = Arr(UBound(Arr) - X)
    Arr(UBound(Arr) - X) = Tmp
Next X
end sub

And of course you need to decode and encode Long Numbers and dates. So:

Public Function EncodeDateGwt(Numero As Double, Optional isDate As Boolean = False) As String

 Dim s As String
Dim a As Double
Dim i As Integer
Dim u As Integer
Dim Base As String
Numero = IIf(isDate, Numero * 1000, Numero)
Base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$"
Do While Val(Numero) <> 0
    a = Numero
    i = 0
    Do While a >= 64
        i = i + 1
        a = a / 64
    Loop
    If i <> u - 1 And u <> 0 Then EncodeDateGwt = EncodeDateGwt & String(u - i - 1, Left(Base, 1))
    a = Int(a)
    EncodeDateGwt = EncodeDateGwt + Mid(Base, a + 1, 1)
    Numero = Numero - a * (64 ^ i)
    u = i
Loop
EncodeDateGwt = EncodeDateGwt & String(i, Left(Base, 1))
End Function

Public Function DecodeDateGwt(Texto As String, Optional isDate As Boolean = False) As Long

Dim Base As String
Dim a As Integer
Base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$"
For a = 1 To Len(Texto)
    DecodeDateGwt = DecodeDateGwt + (InStr(Base, Mid(Texto, a, 1)) - 1) *     (Len(Base) ^ ((Len(Texto) - (a))))
Next
DecodeDateGwt = IIf(isDate, DecodeDateGwt / 1000, DecodeDateGwt)
'devuelve timestamp
End Function

If what you need to encode/decode is a date, then you need to do this before:

    Call encodegwtdate(date2unix("20/02/2016"),true)



Public Function Date2Unix(ByVal vDate As Date) As Long
Date2Unix = DateDiff("s", Unix1970, vDate)
End Function

Public Function Unix2Date(vUnixDate As Long) As Date
Unix2Date = DateAdd("s", vUnixDate, Unix1970)
End Function

Hope you solve it. By the way, does anyone knows what negative numbers means?????

Answer 3

What you are looking at is GWT-RPC serialization format. Unfortunatelly it is not publicly documented. Fortunatelly GWT is open-source so you could look at the source to see how it is produced.

Note: This format might change between GWT versions (I known it did in 2.2). This is most likelly also a reason why Google does not document it - if they did they'd need to keep it backward compatible.

1. Class names that you see are Java classes that Google Checkout uses internally. When GWT is compiled to JS the names get mangled so you don't see them any more.
2. As noted this is GWT-RPC.
3. What you are trying to do is reverse-engineer Google internal APIs. I wouldn't do that because, a. It might change without notice, breaking your app and, b. I'm sure Goog wouldn't like it and it probably violates the service agreement (have you read it?).