0

The transaction start monitoring event of an IIB flow with a MQ Input node produces a base64 encoded byte array of the MQ message. Now I would like to write a Java program that reconstructs this byte array so I can read the headers and body.

The base64 MQ messages looks like this:

TUQgIAIAAAAAAAAACAAAAP////8AAAAAEQEAALgEAABNUUhSRjIgIAQAAAABAAAAQU1RIENFTUJSQSAgICAgIKVV+Fslx7YCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIENFTUJSQSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhbmllbCAgICAgIBYBBRUAAADiboF1+wHSKOpNUf3pAwAAAAAAAAAAAAALICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICALAAAAMC45XGNvbW1vblxqZGtcYmluXGphdmF3LmV4ZTIwMTgxMTI1MTQzNjEyNDcgICAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAA/////1JGSCAAAAACAAAAvAAAAREAAAS4TVFTVFIgICAAAAAAAAAEuAAAACA8bWNkPjxNc2Q+am1zX3RleHQ8L01zZD48L21jZD4gIAAAAEg8am1zPjxEc3Q+cXVldWU6Ly8vTU9OSTwvRHN0PjxUbXM+MTU0MzE1NjU3MjQ1NjwvVG1zPjxEbHY+MjwvRGx2Pjwvam1zPiAAAAAkPHVzcj48VGhlS2V5PlRoZVZhbHVlPC9UaGVLZXk+PC91c3I+PGZvbz5iYXI8L2Zvbz4=

I made following tests to parse this in Java:

import java.io.ByteArrayInputStream;
import java.io.DataInputStream;
import javax.xml.bind.DatatypeConverter;
import org.apache.commons.io.IOUtils;
import org.junit.Assert;
import org.junit.Test;
import com.ibm.mq.headers.CCSID;
import com.ibm.mq.headers.MQHeaderList;
import com.ibm.mq.headers.MQMD;
import com.ibm.mq.headers.MQRFH2;

public class MqMsgTest {

    @Test
    public void allGood() throws Exception {
        String msgBase64 = IOUtils.toString(getClass().getResourceAsStream("/mq-msg.base64"), "UTF-8");
        byte[] msgBytes = DatatypeConverter.parseBase64Binary(msgBase64);
        DataInputStream msgStream = new DataInputStream(new ByteArrayInputStream(msgBytes));
        MQMD mqmd = new MQMD(msgStream);
        Assert.assertEquals("MQHRF2  ", mqmd.getFormat());
        Assert.assertEquals("daniel      ", mqmd.getUserIdentifier());
        MQRFH2 mqrfh2 = new MQRFH2(msgStream);
        Assert.assertEquals("TheValue", mqrfh2.getStringFieldValue("usr", "TheKey"));
        String body = IOUtils.toString(msgStream, CCSID.getCodepage(mqrfh2.nextCharacterSet()));
        Assert.assertEquals("<foo>bar</foo>", body);
    }

    @Test
    public void doesNotWork() throws Exception {
        String msgBase64 = IOUtils.toString(getClass().getResourceAsStream("/mq-msg.base64"), "UTF-8");
        byte[] msgBytes = DatatypeConverter.parseBase64Binary(msgBase64);
        DataInputStream msgStream = new DataInputStream(new ByteArrayInputStream(msgBytes));
        MQHeaderList headers = new MQHeaderList(msgStream, true);
        Assert.assertEquals(2, headers.size());
    }
}

The allGood() test parses the headers and body nicely. But it would fail if the message would not contain a RFH2 header. The doesNotWork() test should parse the headers in a generic way, but it does not work.

How can I parse the base64 encoded MQ message in a flexible way so I have access to the headers and the body?

Daniel Steinmann
  • 2,119
  • 2
  • 15
  • 25

2 Answers2

0

The transaction start monitoring event of an IIB flow with a MQ Input node produces a base64 encoded byte array of the MQ message. Now I would like to write a Java program that reconstructs this byte array so I can read the headers and body.

Why? Why do you combine the MQMD and the MQRFH2 structures then encode it with Base64 only to want to extract it with a Java program?

It sounds like a very poor design.

I dropped your Base64 message into a program that outputs a hex dump of it:

000000: 4D442020 02000000 00000000 08000000  MD  ............
000010: FFFFFFFF 00000000 11010000 B8040000  ????........?...
000020: 4D514852 46322020 04000000 01000000  MQHRF2  ........
000030: 414D5120 43454D42 52412020 20202020  AMQ CEMBRA      
000040: A555F85B 25C7B602 00000000 00000000  ?U?[%??.........
000050: 00000000 00000000 00000000 00000000  ................
000060: 00000000 20202020 20202020 20202020  ....            
000070: 20202020 20202020 20202020 20202020                  
000080: 20202020 20202020 20202020 20202020                  
000090: 20202020 43454D42 52412020 20202020      CEMBRA      
0000a0: 20202020 20202020 20202020 20202020                  
0000b0: 20202020 20202020 20202020 20202020                  
0000c0: 20202020 64616E69 656C2020 20202020      daniel      
0000d0: 16010515 000000E2 6E8175FB 01D228EA  .......?n?u?.?(?
0000e0: 4D51FDE9 03000000 00000000 0000000B  MQ??............
0000f0: 20202020 20202020 20202020 20202020                  
000100: 20202020 20202020 20202020 20202020                  
000110: 0B000000 302E395C 636F6D6D 6F6E5C6A  ....0.9\common\j
000120: 646B5C62 696E5C6A 61766177 2E657865  dk\bin\javaw.exe
000130: 32303138 31313235 31343336 31323437  2018112514361247
000140: 20202020 00000000 00000000 00000000      ............
000150: 00000000 00000000 00000000 01000000  ................
000160: 00000000 00000000 FFFFFFFF 52464820  ........????RFH 
000170: 00000002 000000BC 00000111 000004B8  .......?.......?
000180: 4D515354 52202020 00000000 000004B8  MQSTR   .......?
000190: 00000020 3C6D6364 3E3C4D73 643E6A6D  ... <mcd><Msd>jm
0001a0: 735F7465 78743C2F 4D73643E 3C2F6D63  s_text</Msd></mc
0001b0: 643E2020 00000048 3C6A6D73 3E3C4473  d>  ...H<jms><Ds
0001c0: 743E7175 6575653A 2F2F2F4D 4F4E493C  t>queue:///MONI<
0001d0: 2F447374 3E3C546D 733E3135 34333135  /Dst><Tms>154315
0001e0: 36353732 3435363C 2F546D73 3E3C446C  6572456</Tms><Dl
0001f0: 763E323C 2F446C76 3E3C2F6A 6D733E20  v>2</Dlv></jms> 
000200: 00000024 3C757372 3E3C5468 654B6579  ...$<usr><TheKey
000210: 3E546865 56616C75 653C2F54 68654B65  >TheValue</TheKe
000220: 793E3C2F 7573723E 3C666F6F 3E626172  y></usr><foo>bar
000230: 3C2F666F 6F3E                        </foo>          

The decoded message starts off with the MQMD which is 364 bytes then the MQRFH2 structures which is 202 bytes.

The MQHeaderList class is extremely picky about parsing an MQMessage. The message MUST begin with one of the known MQ classes or it will throw an exception. Using MQHeaderList class for problem messages will probably just lead to more problems.

It would be better to simply dump the message in hex like I did then let the support person or developer figure out what the issue is.

Roger
  • 7,062
  • 13
  • 20
  • Writing monitoring events like this into the database is a design of the IBM integration bus engineers, they call it data capture store. With my tool I would like to read from this store and show whatever is possible from this audited message, in a better human readable format than a hex dump. For me it makes perfect sense. Do you have any idea why MQHeaderList chokes on this message? – Daniel Steinmann Nov 26 '18 at 19:55
  • See my newly posted item. – Roger Nov 26 '18 at 22:37
0

MQHeaderList class is not a magic wand, although, I wish IBM would actually expand its use to include EVERY MQ header. Your problem is that you are assuming MQHeaderList class can do more than what it can. MQHeaderList class can only handle 11 internal MQ headers/structures (aka classes). See here: https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.dev.doc/q030880_.htm

i.e. MQRFH, MQRFH2, MQCIH, MQDLH, MQIIH, MQRMH, MQSAPH, MQWIH, MQXQH, MQDH & MQEPH

As you can see, MQMD is NOT on the list. So, your decoded Base64 message will NOT work with MQHeaderList class.

Also, you should read my comments about MQRFH2 being an embedded message here: Issue While Setting MQRFH2 header in IBM MQ

You have to plug and chug through the data stream that IIB created. Here's some basic code to do the job:

import java.io.ByteArrayInputStream;
import java.io.DataInputStream;
import javax.xml.bind.DatatypeConverter;
import com.ibm.mq.headers.MQMD;
import com.ibm.mq.headers.MQRFH2;

public class Test_IIB_Data
{
   public static void main(String[] args)
   {
      String msgBase64 = "TUQgIAIAAAAAAAAACAAAAP////8AAAAAEQEAALgEAABNUUhSRjIgIAQAAAABAAAAQU1RIENFTUJSQSAgICAgIKVV+Fslx7YCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIENFTUJSQSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhbmllbCAgICAgIBYBBRUAAADiboF1+wHSKOpNUf3pAwAAAAAAAAAAAAALICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICALAAAAMC45XGNvbW1vblxqZGtcYmluXGphdmF3LmV4ZTIwMTgxMTI1MTQzNjEyNDcgICAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAA/////1JGSCAAAAACAAAAvAAAAREAAAS4TVFTVFIgICAAAAAAAAAEuAAAACA8bWNkPjxNc2Q+am1zX3RleHQ8L01zZD48L21jZD4gIAAAAEg8am1zPjxEc3Q+cXVldWU6Ly8vTU9OSTwvRHN0PjxUbXM+MTU0MzE1NjU3MjQ1NjwvVG1zPjxEbHY+MjwvRGx2Pjwvam1zPiAAAAAkPHVzcj48VGhlS2V5PlRoZVZhbHVlPC9UaGVLZXk+PC91c3I+PGZvbz5iYXI8L2Zvbz4=";

      try
      {
         byte[] msgBytes = DatatypeConverter.parseBase64Binary(msgBase64);
         DataInputStream msgStream = new DataInputStream(new ByteArrayInputStream(msgBytes));

         MQMD md = new MQMD(msgStream);

         System.out.println("md.getFormat="+md.getFormat());
         System.out.println("md.getPutApplName="+md.getPutApplName());
         System.out.println("md.getUserIdentifier="+md.getUserIdentifier());

         MQRFH2 rfh2 = new MQRFH2(msgStream);

         System.out.println("rfh2.getFormat="+rfh2.getFormat());
         System.out.println("rfh2.usr.TheKey="+rfh2.getStringFieldValue("usr", "TheKey"));

         int bodyLen = msgBytes.length - rfh2.getStrucLength() - md.size();
         byte[] body = new byte[bodyLen];
         msgStream.read(body);
         System.out.println("body="+new String(body));
      }
      catch (Exception e)
      {
         e.printStackTrace();
      }
   }
}

And when you run it, the output should be:

md.getFormat=MQHRF2  
md.getPutApplName=0.9\common\jdk\bin\javaw.exe
md.getUserIdentifier=daniel      
rfh2.getFormat=MQSTR   
rfh2.usr.TheKey=TheValue
body=<foo>bar</foo>
Roger
  • 7,062
  • 13
  • 20
  • Thank you for your explanation. Your `main()` matches roughly my `allGood()` method in my question. I was looking for a generic way, but it seems not possible. I wonder how RFHUTIL is doing it. Maybe I have to dig through its C source code. – Daniel Steinmann Nov 26 '18 at 22:58
  • Hmm, the source.zip of IH03 Support Pac does not contain the source code for RFHUTIL. And the documentation of RFHUTIL also states that only 5 header types are supported. It seems there is no tool/API available to handle an MQ Message generically. – Daniel Steinmann Nov 27 '18 at 11:19
  • C programs can deal almost trivially with structures like the MQMD - just write the bytes (perhaps converted to characters) straight from the structure. And similarly read on top of a structure. The joys of pointers. Something like "write(fd,&mqmd,sizeof(mqmd))" ... – Mark Taylor Nov 27 '18 at 17:29
  • @Daniel I didn't think the source code was available. Since you are writing it in Java, you have 2 choices: (1) do plug and chug as I mentioned - note: the 1st 4 bytes of the structure will always be the StrucId or – Roger Nov 27 '18 at 17:55
  • (2) You could create your own MQMD class (i.e. MyMQMD) and follow the information here: https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q030970_.htm and then MQHeaderList should work after you register your MyMQMD. See MQHeaderRegistry https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/headers/MQHeaderRegistry.html – Roger Nov 27 '18 at 17:55
  • And don't use anything less than MQ v8 Java JAR files - preferably MQ v9.1 because earlier releases are out of support and also were a little buggy. – Roger Nov 27 '18 at 18:25
  • If you do create your own MQMD class then you can grab my getters and setters from my MDOnly class - I described it here on my blog: https://www.capitalware.com/rl_blog/?p=3537 – Roger Nov 27 '18 at 18:53
  • @MarkTaylor Yes, I saw the C API, but I am restricted to write the code with the IBM MQ Java API; which in theory should work the same as the C API ... – Daniel Steinmann Nov 28 '18 at 13:31
  • @Roger The choice (1) is why I asked this SO question, and (2) I tried but it does not work as well. And yes, I use the 9.1.0.0 jar file, taken from public maven repository. – Daniel Steinmann Nov 28 '18 at 13:36