Retaining message id between 2 MQ brokers using spring integration

Question

I am using a message worfklow made of of message-driven-channel-adapter => channel => outbound-channel-adapter. Its purpose is to transport messages from a MqSeries broker to another MQSeries broker. It's transactionnal (ack required)

The relevant part of it is below (some parts are obvisously missing. If you think they are required, I will edit my post and add them).

My problem is about message headers, and specifically msgId. When I put a message with a messageId in the inbound queue, I expect it to remain the same through the whole pipeline.
But instead the messageId is transformed in the outbound queue, with its content being replaced by a generated ID including outbound queue mananager name.

From the emitter (it's only an exemple for a possible emitting code. I have the same problem from every code I used, as long as I provide a msgId):

com.ibm.mq.MQMessage message = new MQMessage(); 
message.messageId=("TEST MessageId 1234").getBytes();

And from MQExplorer :

From Inbound Queue : MessageId = TEST MessageId 1234
From Outbound Queue : MessageId = AMQ <QM_NAME> <some random(?) code>

There may be a obvious (but not for me) reason, but I don't get it now. I read (well?) that the message Id can be generated by the QM from specific scenarii, or specific commands. But I don't see how it does apply in spring integration.

Any one has an idea on how Spring Integration handles messageId and how I can retain the same through my whole pipeline?

<beans>
        
    <int:channel id="channelMQ_MQ" ></int:channel>

    <!-- Source : MQseries -->
    <!- ... -->
    <bean id="jmsQueue" class="com.ibm.mq.jms.MQQueue" depends-on="jmsConnectionFactory">
        ...       
    </bean>
    <!- ... -->
    <bean id="myListener" class="org.springframework.jms.listener.DefaultMessageListenerContainer" >
        <property name="autoStartup" value="false" />
        <property name="connectionFactory" ref="connectionFactoryCaching" />
        <property name="destination" ref="jmsQueue" />
        <!- ... -->         
        <property name="sessionTransacted" value="true"/>
    </bean>
    
    <int-jms:message-driven-channel-adapter 
        id="jmsIn" 
        container="myListener" 
        channel="channelMQ_MQ" 
        error-channel="processChannel1"/>
                                    

    <!-- Destination MQ_SERIES      -->
        <!- ... -->
    <bean id="jmsQueue2" class="com.ibm.mq.jms.MQQueue" depends-on="jmsConnectionFactory">
        ...
    </bean>
    
    <int-jms:outbound-channel-adapter   channel="channelMQ_MQ" 
                                        id="jmsOut2" 
                                        destination="jmsQueue2" 
                                        connection-factory="connectionFactoryCaching2" 
                                        delivery-persistent="true" 
                                        explicit-qos-enabled="true" 
                                        session-transacted="true" >
    </int-jms:outbound-channel-adapter>

                                        

</beans>

Edit 1:

Following @artem-bilan advice, I set up a header-enricher. But atm, this is not working at all... None of the properties are set up.

    <int:channel id="channel_tmp">
    </int:channel>
    
    
    <int:header-enricher input-channel="channelMQ_MQ"  output-channel="channel_tmp"  id="headerEnricher1">
        <int:header name="MSI" expression="headers.jms_messageId"/>
        <int:header name="JMS_IBM_MQMD_MsgId" expression="headers.jms_messageId"/>
        <int:header name="MSGID" expression="headers.jms_messageId"/>
        <int:header name="MsgId" expression="headers.jms_messageId"/>
        <int:header name="CorrelId" expression="headers.jms_messageId"/>
        <int:header name="GroupId" expression="headers.jms_messageId"/>
        <int:header name="MsggSeqNumber" expression="headers.jms_messageId"/>
        <int:header name="offset" expression="headers.jms_messageId"/>
    </int:header-enricher>
    
    <int-jms:outbound-channel-adapter   channel="channel_tmp" 
                                        id="jmsOut2" 
                                        destination="jmsQueue2" 
                                        connection-factory="connectionFactoryCaching2" 
                                        delivery-persistent="true" 
                                        explicit-qos-enabled="true" 
                                        session-transacted="true" >
    </int-jms:outbound-channel-adapter>

Edit 2 : after some research, we found an IBM doc stating that "To be able to set the Message ID, the JMS destination queue needs to have the property 'MQMD WRITE ENABLE" set to ENABLED. This property allows a JMS application to set the value of the MQMD fields." So we tried to set this property from our JmsQueue :

    <bean id="jmsQueue2" class="com.ibm.mq.jms.MQQueue" depends-on="jmsConnectionFactory">
        ...
      <property name="MQMDWriteEnabled" value="true"></property>
      <property name="MQMDMessageContext" value="2"></property>
    </bean>

Unfortunately, although it was promising, this did not work for the messageId (but other MQMD fields work).

Edit3 :

Following Artem Bilan advice on debugging JmsHeaderMapper, sounds like we found out that byte array is not supported by the header mapper (spring integration version : 5.3.2.RELEASE), but expected by IBM... which leads to header being basically skipped. Thus, this will not work that way:

    <int:header name="JMS_IBM_MQMD_MsgId" expression="headers['jms_messageId'].bytes"/>

Edit 4:

After noticing that current version of spring-integration-jms not accepting "byte[]" type (which is IBM MSGID type), we added a custom header mapper. It worked but we had to retrieved (hex to byte) it from already-mapped message (looking like "ID:3214F1044...") and passing it as a byte array into the header as a "JMS_IBM_MQMD_MsgId" property. And this was a dubious solution because of the triple conversion (MQ [BYTE24] => JMS [ID:String] => Java [Byte[]] => MQ[BYTE24] )
Eventually, we found out that the inbound queue, as well as the outbound one, can be configured such as they will pass all context (jms-mapped headers as well as raw MQ ones). Thus, we don't have to do complex mapping.... only basic mapping (since byte[] are still not mapped in defaultHeaderMapper).

So the final solution is:

<bean id="jmsQueueIN" class="com.ibm.mq.jms.MQQueue" depends-on="jmsConnectionFactory">
  ...
  <property name="MQMDMessageContext" value="2"></property>
  <property name="MQMDReadEnabled" value="true"></property>
</bean>

<bean id="jmsQueueOut" class="com.ibm.mq.jms.MQQueue" depends-on="jmsConnectionFactory">
    ...
    <property name="MQMDWriteEnabled" value="true"></property>
    <property name="MQMDMessageContext" value="2"></property>       
</bean>


<bean id="mqCompatibleJmsHeaderMapper" class="com.my.company.mappers.MqCompatibleJmsHeaderMapper"/> 
<int-jms:outbound-channel-adapter   channel="channel_MQ_MQ" 
                                    id="jmsOut" 
                                    destination="jmsQueueOUT" 
                                    ... 
                                    header-mapper="mqCompatibleJmsHeaderMapper">
...                                     
</int-jms:outbound-channel-adapter>

_

public class MqCompatibleJmsHeaderMapper extends DefaultJmsHeaderMapper {
...
  public void fromHeaders(MessageHeaders headers, Message jmsMessage) {
    Object messageId = headers.get(WMQConstants.JMS_IBM_MQMD_MSGID);
    if(messageId !=null) {
        if (messageId instanceof byte[]) {
            jmsMessage.setObjectProperty(WMQConstants.JMS_IBM_MQMD_MSGID, messageId);
        }else  {
         ...
        }
    }
    super.fromHeaders(headers, jmsMessage);
  }

...
}

The best way is to debug your flow to see what is going on with your `messageId` property. The `` is based on the `DefaultJmsHeaderMapper` which maps any arbitrary JMS property like this: `mapArbitraryProperty(jmsMessage, headers, propertyName);`. To debug I would suggest to use a global `` for the `` which is going to log how your messages are traveling in Spring Integration flow with all their content. Check if your `messageId` is present there in headers. — Artem Bilan, Apr 12 '22 at 14:39
See answer [How to set message Id for IBM MQ using java program](https://stackoverflow.com/a/52890449/638413) — Daniel Steinmann, Apr 12 '22 at 14:58
@artem-bilan I already added an interceptor (I guess it does the job) thus I used it to look into at what happened "afterSendCompletion". And I saw my message Id associated with "jms_messageId" (in a hex format, like that : jms_messageId=ID:335f54455354204d65737361676549642031323337383936). It may not be in the correct field, if I follow daniel-steinmann hint. But if I look in IBM Explorer, it looks like it fits in the right field (or does MQE show untrustworthy data?). — Marvin, Apr 13 '22 at 06:52
@daniel-steinmann I tried with this header, but i got a an error "MQRC_PROPERTY_NAME_ERROR". Also, my goal is to maintain it from Broker A to Broker B, not to set it (unless this is related). Whatever the method for providing it is (the code above is just an example) — Marvin, Apr 13 '22 at 06:56
So, your question is then how to get rid of that `jms_` prefix and prop will be propagated correctly? — Artem Bilan, Apr 13 '22 at 11:15
It didn't seen that obvious to me.... do you mean that the behavior is "normal" and that I should use a headerMapper to change it, ie remove the "jms_" prefix? — Marvin, Apr 13 '22 at 12:53

score 1 · Answer 1 · answered Apr 13 '22 at 13:48

1

It didn't seen that obvious to me.... do you mean that the behavior is "normal" and that I should use a headerMapper to change it, ie remove the "jms_" prefix?

Well, that header is mapped in the DefaultJmsHeaderMapper like this: headers.put(JmsHeaders.MESSAGE_ID, messageId);. So, it indeed comes as a jms_messageId. And it is really not mapped on the outbound side:

if (StringUtils.hasText(headerName) &&
                    !headerName.startsWith(JmsHeaders.PREFIX) &&
                    jmsMessage.getObjectProperty(headerName) == null) {

I think there was a reason to ignore them since not all JMS vendors allows to override all those org.springframework.jms.support.JmsHeaders.

For your use-case you can do this before <int-jms:outbound-channel-adapter>:

<header-enricher>
    <header name="messageId" expression="headers.jms_messageId"/>
</header-enricher>

answered Apr 13 '22 at 13:48

Artem Bilan

113,505
11
91
118

Thank you for this clue. Unfortunately, this is not working. I think I may have misses something in the pattern , because it sounds like the right solution . I edited my post with what I have tried. I did not manage to modify any of the outbound message MQ headers – Marvin Apr 14 '22 at 14:57
According to this discussion (https://stackoverflow.com/questions/52889361/how-to-set-message-id-for-ibm-mq-using-java-program) the header name must be `JMS_IBM_MQMD_MsgId`. However I see that you have already one in your `header-enricher`. Can you double check, please, what type is your `headers.jms_messageId`? There are limited types which can be mapped to JMS properties: `Boolean.class, Byte.class, Double.class, Float.class, Integer.class, Long.class, Short.class, String.class` – Artem Bilan Apr 14 '22 at 15:15
The property message.getHeaders().get("jms_messageId").getClass()is a "String" whether I set it from a Java code or from IBM Mq test message (amqsput). I read it from my interceptor (on the first channel) . The description from IBM doc states that is is a "byte string". Also the new Id is obviously generated (since it holds the QM name in it), so it doesn't look like a pb of conversion (but I may be wrong). In case MQExplorer doesn't conceals missing headers, I tried to check them with tcpdump, with no success : only one occurence of the messageId can be seen though I set it multiple times – Marvin Apr 14 '22 at 16:12
we tried a new solution suggested by IBM doc (my 2nd edit), but messageId remained changed everytime it reaches message broker. Could there be some override from within spring-integration-jms ? I checked source code, but couldn't find anyting relevant regarding this point – Marvin Apr 27 '22 at 14:53
1

I would suggest to debug `DefaultJmsHeaderMapper.fromHeaders()` to see what is going on with your `messageId` header. Then down to the `JmsTemplate.doSend()` and its `producer.send()` call. – Artem Bilan Apr 27 '22 at 15:01
I followed you advice: sounds like it lead to a clue, although I am unsure what I could do now. I saw (previously not loggued) a warning saying "WARN o.s.i.jms.DefaultJmsHeaderMapper - failed to map Message header 'JMS_IBM_MQMD_MsgId' to JMS property com.ibm.msg.client.jms.DetailedMessageFormatException: JMSCC0051: property 'JMS_IBM_Report_COA' should be set using type '[B', not 'java.lang.String'." . Through DefaultJmsHeaderMapper.fromHeaders(), I saw that byte[].class is not part of SUPPORTED_PROPERTY_TYPES... though IBM is expecting an array of byte. – Marvin Apr 27 '22 at 19:08
1

Yeah... Looks like `byte[]` is even supported by ActiveMQ. Perhaps we need to revise `DefaultJmsHeaderMapper` to include that, too. but again: how about sending a `messageId` header instead of `JMS_IBM_MQMD_MsgId` ? – Artem Bilan Apr 27 '22 at 19:22
1

You can extend a `DefaultJmsHeaderMapper` and override its `fromHeaders()` to to not call a `super` when you got a `JMS_IBM_MQMD_MsgId` header and populate it yourself. – Artem Bilan Apr 27 '22 at 19:24
The override seems a good way, yes. Will try it immediately and come back with it. Regarding the messageId Header, I actually don't understand your point : if the problem is the unsupported type, and since "messageId" (or "msgId") will be (mis)parsed the same way, how should this work? – Marvin Apr 27 '22 at 21:15
I don't know: maybe IBM JMS client is able to convert it properly to the `JMS_IBM_MQMD_MsgId` just before placing the message onto the network. Or IBM MQ Broker can parse it in the end already... – Artem Bilan Apr 27 '22 at 21:37
after many tests, and additionnal research, following your many hints, we ended up finding what looks like the right solution ("Edit 4"). Not sure it is the cleanest one, but it prooved resilient. Without your many helps, I would not have been able te find this out. Thus : thanks a lot! – Marvin May 04 '22 at 08:04
@Marvin you should accept the answer or write your own. – JoshMc May 04 '22 at 08:39
@JoshMc ok doing that – Marvin May 04 '22 at 10:19

score 1 · Accepted Answer · answered May 04 '22 at 10:15

This answer computes the various (and very usefull!) comments and answers that helped solve our problem.

After noticing that current version of spring-integration-jms does not accept "byte[]" type (which is IBM MSGID type), we added a custom header mapper.

The original RAW msgId was not available from the inbound : instead, we received JMS-mapped heades such as :

JMSXAppID=com.my.company.test.MqProducer
jms_replyTo=queue://QM2/QUEUE.OUT.MOBA?targetClient=1 jms_correlationId=ID:4142434423313233343536373839305f3100000000000000 jms_messageId=ID:4142434423313233343536373839305f3100000000000000

They were received this way from the listner attached with the inbound adapter.
Thus, the first draft used a headerMapper to transform jmsMessageId (Id:...) into IBM Mq BYTE24 comptabile (thus : byte[] into "JMS_IBM_MQMD_MsgId" property)

But this was a hazardous solution because of the triple conversion (MQ [BYTE24] => JMS [ID:String] => Java [Byte[]] => MQ[BYTE24] )

Eventually, we found out that the inbound queue, as well as the outbound one, can be configured such as they will pass all context (jms-mapped headers as well as raw MQMD ones) : using MQMDMessageContext = CMQC.MQPMO_SET_ALL_CONTEXT (2) and MQMDRead/WriteEnabled = true (according to whether it is the inbound or outbound queue). This way, all required fields were available from start :

JMS_IBM_MQMD_PutApplName=CustomOwnApplName
JMSXAppID=com.my.company.test.MqProducer
JMS_IBM_MQMD_ReplyToQ=
jms_replyTo=queue://QM2/QUEUE.OUT.MOBA?targetClient=1
JMS_IBM_MQMD_CorrelId=[B@47e0d39f
jms_correlationId=ID:4142434423313233343536373839305f3100000000000000
JMS_IBM_MQMD_MsgId=[B@399141ee
jms_messageId=ID:4142434423313233343536373839305f3100000000000000

Thus, we don't have to do dubious mapping.... only basic mapping (since byte[] are still not mapped in defaultHeaderMapper).

So the final solution is:

<bean id="jmsQueueIN" class="com.ibm.mq.jms.MQQueue" depends-on="jmsConnectionFactory">
  ...
  <property name="MQMDMessageContext" value="2"></property>
  <property name="MQMDReadEnabled" value="true"></property>
</bean>

<bean id="jmsQueueOut" class="com.ibm.mq.jms.MQQueue" depends-on="jmsConnectionFactory">
    ...
    <property name="MQMDWriteEnabled" value="true"></property>
    <property name="MQMDMessageContext" value="2"></property>       
</bean>


<bean id="mqCompatibleJmsHeaderMapper" class="com.my.company.mappers.MqCompatibleJmsHeaderMapper"/> 
<int-jms:outbound-channel-adapter   channel="channel_MQ_MQ" 
                                    id="jmsOut" 
                                    destination="jmsQueueOUT" 
                                    ... 
                                    header-mapper="mqCompatibleJmsHeaderMapper">
...                                     
</int-jms:outbound-channel-adapter>

_

public class MqCompatibleJmsHeaderMapper extends DefaultJmsHeaderMapper {
...
  public void fromHeaders(MessageHeaders headers, Message jmsMessage) {
    Object messageId = headers.get(WMQConstants.JMS_IBM_MQMD_MSGID);
    if(messageId !=null) {
        if (messageId instanceof byte[]) {
            jmsMessage.setObjectProperty(WMQConstants.JMS_IBM_MQMD_MSGID, messageId);
        }else  {
         ...
        }
    }
    super.fromHeaders(headers, jmsMessage);
  }

...
}

Please, raise a GH issue, so we will add a `byte[].class` as a valid type for JMS headers mapping. — Artem Bilan, May 04 '22 at 14:33
done : https://github.com/spring-projects/spring-integration/issues/3788 — Marvin, May 05 '22 at 12:16

Retaining message id between 2 MQ brokers using spring integration

2 Answers2