rocketmq throw exception "[TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while"

Question

version:rocketmq-all-4.1.0-incubating

We send msg 1000 QPS，sync send, but throw exception:-

[TIMEOUT_CLEAN_QUEUE] broker busy, start flow control for a while

There is the related code:

while (true) {
        try {
            if (!this.brokerController.getSendThreadPoolQueue().isEmpty()) {
                final Runnable runnable = this.brokerController.getSendThreadPoolQueue().peek();
                if (null == runnable) {
                    break;
                }
                final RequestTask rt = castRunnable(runnable);
                if (rt == null || rt.isStopRun()) {
                    break;
                }

                final long behind = System.currentTimeMillis() - rt.getCreateTimestamp();
                if (behind >= this.brokerController.getBrokerConfig().getWaitTimeMillsInSendQueue()) {
                    if (this.brokerController.getSendThreadPoolQueue().remove(runnable)) {
                        rt.setStopRun(true);
                        rt.returnResponse(RemotingSysResponseCode.SYSTEM_BUSY, String.format("[TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: %sms, size of queue: %d", behind, this.brokerController.getSendThreadPoolQueue().size()));
                    }
                } else {
                    break;
                }
            } else {
                break;
            }
        } catch (Throwable ignored) {
        }
    }
}

I find broker the default value of sendMessageThreadPoolNums is 1，

/**
 * thread numbers for send message thread pool, since spin lock will be used by default since 4.0.x, the default value is 1.
 */
private int sendMessageThreadPoolNums = 1; //16 + Runtime.getRuntime().availableProcessors() * 4;
private int pullMessageThreadPoolNums = 16 + Runtime.getRuntime().availableProcessors() * 2;

but the previous version isn't 1, and if I configure sendMessageThreadPoolNums = 100, can resolve this question ? It will lead to what is different with default value. thanks

JaskeyLam · Answer 1 · 2018-03-28T09:11:18.603

SHORT ANSWER:

you have two choices:

set sendMessageThreadPoolNums to a small number, say 1, which is the default value after version 4.1.x. And, remain the default value of useReentrantLockWhenPutMessage=false, which is introduced after 4.1.x

 sendMessageThreadPoolNums=1 
 useReentrantLockWhenPutMessage=false

If you need to use a large numbers of threads to process sending message, you'd better use useReentrantLockWhenPutMessage=true

 sendMessageThreadPoolNums=128//large thread numbers
 useReentrantLockWhenPutMessage=true  // indicating that do NOT use spin lock but use ReentrantLock when putting message

rocketmq throw exception "[TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while"

1 Answers1