0

So I have a jboss AMQ 6.3 cluster (3 pods in OpenShift) that keeps growing daily. Looking in the Java Console, there are a Lot of Queues named clever things like

error-user006107e3-85ce-bdf1-f414-34fc448c1eb4

which I'm assuming is the reason KahaDB keeps growing in size (currently 120mb each split, up from 100 last week...) Is there a way to clean these likely dead queues to stop us from running out of memory every 2 weeks?

Edit: Not so sure the error-user is the villain here.. But it does keep growing..

Edit2: Here's the activemq.xml

<beans xmlns="http://www.springframework.org/schema/beans" 
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
       xsi:schemaLocation="http://www.springframework.org/schema/beans    
       http://www.springframework.org/schema/beans/spring-beans.xsd 
           http://activemq.apache.org/schema/core    
           http://activemq.apache.org/schema/core/activemq-core.xsd"> 
    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>file:${activemq.conf}/credentials.properties</value>
        </property>
    </bean>
    <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
          lazy-init="false" scope="singleton"
          init-method="start" destroy-method="stop">
    </bean>
    <broker xmlns="http://activemq.apache.org/schema/core"    brokerName="${HOSTNAME}" dataDirectory="${activemq.data}" schedulePeriodForDestinationPurge="10000">
        <destinationPolicy>
            <policyMap>
                <policyEntries>
                    <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb" maxBrowsePageSize="700" gcInactiveDestinations="true" inactiveTimoutBeforeGC="30000">

                        <networkBridgeFilterFactory>
                            <conditionalNetworkBridgeFilterFactory replayWhenNoConsumers="true" replayDelay="500" />
                        </networkBridgeFilterFactory>
                    </policyEntry>
                    <policyEntry topic=">" producerFlowControl="true" gcInactiveDestinations="true" inactiveTimoutBeforeGC="30000">
                        <pendingMessageLimitStrategy>
                            <constantPendingMessageLimitStrategy limit="1000" />
                        </pendingMessageLimitStrategy>
                    </policyEntry>
                </policyEntries>
            </policyMap>
        </destinationPolicy>
        <managementContext>
            <managementContext createConnector="false" />
        </managementContext>

        <ioExceptionHandler>
            <defaultIOExceptionHandler ignoreNoSpaceErrors="false"/>
        </ioExceptionHandler>

        <networkConnectors>
            <networkConnector userName="userName" password="password" uri="dns://amq-cluster:61616/?transportType=tcp&amp;queryInterval=30" messageTTL="-1" consumerTTL="1" />
        </networkConnectors>
        <persistenceAdapter>
            <kahaDB directory="${activemq.data}/kahadb" />
        </persistenceAdapter>

        <plugins>
            <jaasAuthenticationPlugin configuration="activemq" />
        </plugins>
        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage percentOfJvmHeap="70" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="100 gb" />
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="50 gb" />
                </tempUsage>
            </systemUsage>
        </systemUsage>
        <transportConnectors>
            <transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600&amp;transport.hbGracePeriodMultiplier=2.5" />
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600" />
        </transportConnectors>
        <shutdownHooks>
            <bean xmlns="http://www.springframework.org/schema/beans"    class="org.apache.activemq.hooks.SpringContextHook" />
        </shutdownHooks>
    </broker>
    <!-- Do not expose the console or other webapps
    <import resource="jetty.xml" />
    -->
</beans>
user3258911
  • 159
  • 5
  • 13
  • Do you have a DLQ or similar? It sounds like messages are being sent to a queue with no consumer to remove them. you would need to provide some config details at minimum and some queue stats. – user3788685 Mar 01 '21 at 11:34
  • @user3788685 Updated question with config.Absolutely. How do I pull queue stats that are relevant? – user3258911 Mar 01 '21 at 11:45
  • have you browsed the queues with jmx or the amq web console to see what they are actually gathering up? that should help narrow it down. What is your producer? – user3788685 Mar 01 '21 at 15:51
  • Digging through a heap dump, there are thousands upon thousands of StoreUsage objects named "default:store:queue://error-user{uuid}:store – user3258911 Mar 10 '21 at 09:39
  • your config does not show the cleanupinterval for KahabDB. Look here https://activemq.apache.org/kahadb – user3732793 Mar 16 '21 at 10:30
  • cleanupInterval is set to 30000 by default. Setting it explicitly did not change anything. – user3258911 Mar 18 '21 at 14:06

0 Answers0