ActiveMQ is not shutting down properly -INFO: Regular shutdown not successful, sending SIGKILL to process

Question

Whenever I try to stop ActiveMQ installed on RHEL server, it doesn't stops gracefully. Does anyone knows why? I am not sure why it tries to connect to JMX broker as shown below and fails. What do I need to fix these issues?

[activemq@myserver apache-activemq-5.15.11]$ bin/activemq stop
INFO: Loading '/web/servers/apache-activemq-5.15.11//bin/env'
INFO: Using java '/bin/java'
INFO: Waiting at least 30 seconds for regular process termination of pid '2963' :
Java Runtime: Oracle Corporation 1.8.0_252 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
  Heap sizes: current=62976k  free=61991k  max=932352k
    JVM args: -Xms64M -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/web/servers/apache-activemq-5.15.11//conf/login.config -Dactivemq.classpath=/web/servers/apache-activemq-5.15.11//conf:/web/servers/apache-activemq-5.15.11//../lib/: -Dactivemq.home=/web/servers/apache-activemq-5.15.11/ -Dactivemq.base=/web/servers/apache-activemq-5.15.11/ -Dactivemq.conf=/web/servers/apache-activemq-5.15.11//conf -Dactivemq.data=/web/servers/apache-activemq-5.15.11//data
Extensions classpath:
  [/web/servers/apache-activemq-5.15.11/lib,/web/servers/apache-activemq-5.15.11/lib/camel,/web/servers/apache-activemq-5.15.11/lib/optional,/web/servers/apache-activemq-5.15.11/lib/web,/web/servers/apache-activemq-5.15.11/lib/extra]
ACTIVEMQ_HOME: /web/servers/apache-activemq-5.15.11
ACTIVEMQ_BASE: /web/servers/apache-activemq-5.15.11
ACTIVEMQ_CONF: /web/servers/apache-activemq-5.15.11/conf
ACTIVEMQ_DATA: /web/servers/apache-activemq-5.15.11/data
Connecting to pid: 2963
INFO: failed to resolve jmxUrl for pid:2963, using default JMX url
Connecting to JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
INFO: Broker not available at: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
...............................
INFO: Regular shutdown not successful,  sending SIGKILL to process
INFO: sending SIGKILL to pid '2963'

EDIT:

Adding activemq.xml configuration file

   <!--
    Licensed to the Apache Software Foundation (ASF) under one or more
    contributor license agreements.  See the NOTICE file distributed with
    this work for additional information regarding copyright ownership.
    The ASF licenses this file to You under the Apache License, Version 2.0
    (the "License"); you may not use this file except in compliance with
    the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-->
<!-- START SNIPPET: example -->
<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

    <!-- Allows us to use system properties as variables in this configuration file -->
    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>file:${activemq.conf}/credentials.properties</value>
        </property>
    </bean>

   <!-- Allows accessing the server log -->
    <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
          lazy-init="false" scope="singleton"
          init-method="start" destroy-method="stop">
    </bean>

    <!--
        The <broker> element is used to configure the ActiveMQ broker.
    -->
    <broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}">

        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" >
                    <!-- The constantPendingMessageLimitStrategy is used to prevent
                         slow topic consumers to block producers and affect other consumers
                         by limiting the number of messages that are retained
                         For more information, see:

                         http://activemq.apache.org/slow-consumer-handling.html

                    -->
                  <pendingMessageLimitStrategy>
                    <constantPendingMessageLimitStrategy limit="1000"/>
                  </pendingMessageLimitStrategy>
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>


        <!--
            The managementContext is used to configure how ActiveMQ is exposed in
            JMX. By default, ActiveMQ uses the MBean server that is started by
            the JVM. For more information, see:

            http://activemq.apache.org/jmx.html
        -->
        <managementContext>
            <managementContext createConnector="false"/>
        </managementContext>

        <!--
            Configure message persistence for the broker. The default persistence
            mechanism is the KahaDB store (identified by the kahaDB tag).
            For more information, see:

            http://activemq.apache.org/persistence.html
        -->
        <persistenceAdapter>
            <kahaDB directory="${activemq.data}/kahadb"/>
        </persistenceAdapter>


          <!--
            The systemUsage controls the maximum amount of space the broker will
            use before disabling caching and/or slowing down producers. For more information, see:
            http://activemq.apache.org/producer-flow-control.html
          -->
          <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage percentOfJvmHeap="70" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="100 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="50 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

        <!--
            The transport connectors expose ActiveMQ over a given protocol to
            clients and other brokers. For more information, see:

            http://activemq.apache.org/configuring-transports.html
        -->
        <transportConnectors>
            <!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
            <transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
            <transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
            <transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
            <transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
        </transportConnectors>

        <!-- destroy the spring context on shutdown to stop jetty -->
        <shutdownHooks>
            <bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
        </shutdownHooks>

    </broker>

    <!--
        Enable web consoles, REST and Ajax APIs and demos
        The web consoles requires by default login, you can disable this in the jetty.xml file

        Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
    -->
    <import resource="jetty.xml"/>

</beans>
<!-- END SNIPPET: example -->

Adding activemq.log file from /web/servers/apache-activemq-5.15.11/data

Pasting shareable link below:

https://drive.google.com/file/d/1vQ6HkOu53mzMi-GqbGxaQP6EJNC4RT4G/view?usp=sharing

@JustinBertram Please find my updated post with the details you requested. I had to put GD link since logs were exceeding 30k limit of the post. Let me know if you need more info. Thanks ! — Tan, Jul 02 '20 at 16:43
Are you executing the script as the same user who started the process? — yegodm, Jul 02 '20 at 17:08
@yegodm So basically, I log into the RHEL server using my credentials and then switch to another user (`su - activemq`) which is `activemq`. My network team created this user for me so that I can modify files and folders inside apache acrivemq . Could this be an issue? — Tan, Jul 02 '20 at 17:55
No, I don't think so. If the broker runs under `activemq` user, should be fine. — yegodm, Jul 02 '20 at 17:57
Maybe this will help: https://stackoverflow.com/a/19175470/1654233 — yegodm, Jul 02 '20 at 18:02
Thanks. It looks like that person is having issues accessing activemq from the browser based on this line `When I try to connect with Apache ActiveMQ Browser the connections keeps failing`. Mine is fine. — Tan, Jul 02 '20 at 18:21
Were you ever able to resolve this? I appear to be having the same issue of activemq not shutting down regardless of how long it is given via ACTIVEMQ_KILL_MAXSECONDS — Ben Pennell, Nov 17 '20 at 19:46
@BenPennell Not yet. If you are able to, please let me know. It's surprising that on one RHEL server, it doesn't complain but in other it does complain. My ActiveMQ shuts down though after a min or two. — Tan, Nov 17 '20 at 21:04

score 0 · Answer 1 · answered Nov 20 '20 at 20:36

0

Don't know if this is the issue for you, but after some reading it sounded like activemq uses jmx for its shutdown hook, but the default config that ships with activemq does not setup the jmx connector.

For me, activemq was able to shutdown if I changed:

        <managementContext>
            <managementContext createConnector="false"/>
        </managementContext>

To specify a jmx connectorPort:

        <managementContext>
            <managementContext connectorPort="1099"/>
        </managementContext>

Some references:

answered Nov 20 '20 at 20:36

Ben Pennell

449
1
3
13

Thank you. I will take a look. Appreciated. Is 1099 port default port to be used by everyone? – Tan Nov 20 '20 at 20:53
Its the default jmx port, and is the port that activemq logs during startup or shutdown for me. – Ben Pennell Nov 21 '20 at 21:33

ActiveMQ is not shutting down properly -INFO: Regular shutdown not successful, sending SIGKILL to process

1 Answers1