2

when i send message to broker,this exception occasionally occurs. 

MQBrokerException: CODE: 2  DESC: [TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while

This means broker is too busy(when tps>1,5000) to handle so many sending message request. What would be the most impossible reason to cause this? Disk ,cpu or other things? How can i fix it?

allan
  • 21
  • 3

2 Answers2

2

There are many possible ways.

The root cause is that, there are some messages has waited for long time and no worker thread processes them, rocketmq will trigger the fast failure.

So the below is the cause:

  1. Too many thread are working and they are working very slow to process storing message which makes the cache request is timeout.

  2. The jobs it self cost a long time to process for message storing.

This may be because of:

2.1 Storing message is busy, especially when SYNC_FLUSH is used.

2.2 Syncing message to slave takes long when SYNC_MASTER is used.

JaskeyLam
  • 15,405
  • 21
  • 114
  • 149
  • I am also facing the same issue sometimes.In my broker config, brokerRole = ASYNC_MASTER and flushDiskType = ASYNC_FLUSH. I have a producer and it is trying to send 500 data parallely.But after sending some data(100/150), getting the above exception.Some times getting No route info find exception also. – Santanu Mar 21 '18 at 08:39
2

In /broker/src/main/java/org/apache/rocketmq/broker/latency/BrokerFastFailure.java you can see:

final long behind = System.currentTimeMillis() - rt.getCreateTimestamp();
if (behind >= this.brokerController.getBrokerConfig().getWaitTimeMillsInSendQueue()) {
  if (this.brokerController.getSendThreadPoolQueue().remove(runnable)) {
    rt.setStopRun(true);
    rt.returnResponse(RemotingSysResponseCode.SYSTEM_BUSY, String.format("[TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: %sms, size of queue: %d", behind, this.brokerController.getSendThreadPoolQueue().size()));
  }
}

In common/src/main/java/org/apache/rocketmq/common/BrokerConfig.java, getWaitTimeMillsInSendQueue() method returns

public long getWaitTimeMillsInSendQueue() {
    return waitTimeMillsInSendQueue;
}

The default value of waitTimeMillsInSendQueue is 200, thus you can just set it bigger to make the queue waiting for longer time. But if you wanna solve the problem completely, you should follow Jaskey's advice and check your code.

Wayne Sudo
  • 92
  • 5