ActiveMQ - After a long time running, broker accepts messages but doesn't enqueue anything

Question

My configuration is:

• ActiveMQ 5.15.1
• atomikos transactions jms 4.0.6
• Spring boot 2.1.7.RELEASE

My application is sending messages by Spring JmsTemplate.convertAndSend method :

import org.springframework.jms.core.JmsTemplate;

@Component
@Slf4j
public class AsynchronousMajProducer {

  @Autowired
  private ActiveMQConfig activeMQConfig;
  @Autowired
  private JmsTemplate jmsTemplate;

  private void convertAndSend(String msg) {
     jmsTemplate.convertAndSend(activeMQConfig.privateQueue(), msg, message -> {
         String[] versionParts = buildProperties.getVersion().split("\\.");
         String version = versionParts[0] + "." + versionParts[1];
         message.setStringProperty("VERSION", version);
         message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, 2000);
         return message;
     });
  }
}

After a long time running (multiple weeks), some queues doesn't enqueue messages anymore (and some works well).

For those failing, the convertAndSend method doesn't throw any exceptions, all seems to be OK, but finally, nothing is enqueued ... (enqueue count on AMQ console never increases).

Below is an AMQ log on a successful sending :

2019-11-22 09:37:29,647 | INFO  | Sending message: ActiveMQTextMessage {commandId = 8, responseRequired = false, 
messageId = ID:NMI16510-60675-1574375565048-1:23:2:1:1, originalDestination = null, originalTransactionId = null, 
producerId = ID:NMI16510-60675-1574375565048-1:23:2:1, destination = queue://testportail, 
transactionId = XID:[1096044365,globalId=3137322e31372e36352e37372e746d313537343337353834383839343030343731,
branchId=3137322e31372e36352e37372e746d343732], expiration = 1574375885026, timestamp = 1574375849026, 
arrival = 0, brokerInTime = 0, brokerOutTime = 0, correlationId = null, replyTo = null, persistent = 
true, type = null, priority = 9, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, 
userID = null, content = org.apache.activemq.util.ByteSequence@15d17509, marshalledProperties = org.apache.activemq.util.ByteSequence@34beb533, 
dataStructure = null, redeliveryCounter = 0, size = 1122, properties = {VERSION=2.0, AMQ_SCHEDULED_DELAY=2000}, 
readOnlyProperties = false, readOnlyBody = false, droppable = false, jmsXGroupFirstForConsumer = false, 
text = {
  "type" : "MAJ_CONTACT",
  "id" : 124
}} | org.apache.activemq.broker.util.LoggingBrokerPlugin | ActiveMQ Transport: tcp:///172.17.65.77:61220@61611

2019-11-22 09:37:32,001 | DEBUG | Firing: Job [id=ID:NMI16510-60675-1574375565048-1:23:2:1:1, startTime=Fri Nov 22 09:37:30 NCT 2019, 
delay=2000, period=0, repeat=0, nextTime=Fri Nov 22 09:37:32 NCT 2019, executionCount = 1] | org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl 
| JobScheduler:JMS

2019-11-22 09:37:32,002 | DEBUG | Set message ID:NMI16510-60675-1574375565048-1:23:2:1:1 timestamp from 1574375849026 to 1574375852002 
| org.apache.activemq.broker.scheduler.SchedulerBroker | JobScheduler:JMS

2019-11-22 09:37:32,003 | DEBUG | ACTIVEMQ Message ID:NMI16510-60675-1574375565048-1:23:2:1:1 sent to queue://testportail 
| org.apache.activemq.broker.region.Queue | JobScheduler:JMS

When it fails, I only get the first log Sending message. No ERROR or WARN messages on AMQ log. I noticed some queues are affected and some not.

To 'fix' the problem, I have to :

1. Stop the broker
2. Delete this folder : /apache-activemq-5.15.1/data/ACTIVEMQ/scheduler
3. Restart the broker

In production, we do it every day to anticipate this problem. It's not a good solution, because messages previously send by the application but not enqueued are definitly lost.

Is it a known issue of AMQ scheduler ? Is it a bad/good practice do restart AMQ frequently ?

Answer 1

I've just found a relative issue but for Topic here : [http://activemq.2283324.n4.nabble.com/ActiveMQ-Topic-Messages-Not-Delivered-td4703502.html] It has been resolved by touching to storeUsage and tempUsage in activemq.xml :

Original
            <storeUsage>
                <storeUsage limit="100 gb"/>
            </storeUsage>
            <tempUsage>
                <tempUsage limit="50 gb"/>
            </tempUsage>
Modification
            <storeUsage>
                <storeUsage limit="20 mb"/>
            </storeUsage>
            <tempUsage>
                <tempUsage limit="20 mb"/>
            </tempUsage>

The problem was a storage issue for kahadb wich was in a smaller storage than 100Go.

Thanks to swone!