1

I have a situation where a RHEL 6.4 server is sending TERM and KILL signals too quickly, before applications and databases have had a chance to gracefully stop. It seems that Upstart is handing control to the sysv-rc scripts prematurely.

To troubleshoot this, I've tried adding sleep and logger commands into the Upstart config. The pre-script stanza is writing to syslog, but the sleep never completes because the system reboots within 10 seconds. I've also added a kill timeout which is ignored. 

# cleanup at system shutdown
# Does stop all on apps, stops monit instances, then does a clean.
start on runlevel [016]

console output
kill timeout 120

task

pre-start script
    logger -s -t "arcsight-services-stopall" "Running pre-start..."
    /etc/init.d/arcsight_services stop
    sleep 60
end script

script
    logger -s -t "arcsight-services-stopall" "Running script..."
    /etc/init.d/arcsight_services shutdown monit
    /etc/init.d/arcsight_services clean all
end script

I know Upstart was suppose to parallelize the boot/halt process, but it has only paralyzed my debugging attempts.

Under RHEL 6, what is the definitive order of scripts that get executed after issuing: shutdown -r now?

  1. shutdown -r now
  2. ?????
  3. /etc/init/rc.conf (Upstart)
  4. /etc/rc.d/rc (sysv-rc)
  5. ?????
  6. /etc/rc3.d/K* (sysv-rc)
  7. /etc/rc6.d/S* (sysv-rc)
  8. ?????

Where do the other /etc/init/*.conf scripts get called?

UPDATE: In dissecting /etc/rc.d/rc, I found that if I touch /var/run/confirm, the process enters an interactive mode and then my sleep & logger commands seem to execute. This baffles me, because I thought, at that point, Upstart has passed control to sysv-rc.

skohrs
  • 1,520
  • 11
  • 23
  • Don't write upstart scripts on RHEL 6. They are not supported (even though it actually uses upstart). Use a SysV style init script instead. – Michael Hampton Dec 23 '13 at 21:29
  • HP ArcSight Developers, are you reading this? "Don't write upstart scripts on RHEL 6." Thanks @MichaelHampton. – skohrs Dec 26 '13 at 13:38
  • 1
    It's really just an unfortunate artifact of timing that RHEL ended up with Upstart at all. It was tried in Fedora and quickly abandoned since it was crap, but RHEL 6 went into feature freeze at just that time. [Virtually everyone else has abandoned Upstart, too](http://serverfault.com/a/518906/126632), and RHEL 7 uses systemd. That will last a little longer... – Michael Hampton Dec 26 '13 at 16:27

1 Answers1

1

While this answer doesn't address the order of scripts that get executed during a RHEL 6 shutdown, it does solve the issue of the system killing processes before they are gracefully stopped.

/etc/init/arcsight-services-stopall.conf:

# cleanup at system shutdown
# Does stop all on apps, stops monit instances, then does a clean.

start on starting rc RUNLEVEL=[016]

task
kill timeout 330

pre-start script
    logger -s -t "ArcSight" "ArcSight ESM shutdown initiated..."
    /etc/init.d/arcsight_services shutdown all
    /etc/init.d/arcsight_services shutdown monit
    /etc/init.d/arcsight_services clean all
    sleep 300
end script

script
    logger -s -t "ArcSight" "ArcSight ESM shutdown complete."
end script

The key was modifying start on starting rc RUNLEVEL=[016]. By starting this script while /etc/init/rc.conf is just starting, it blocks for 5 minutes, before the sysv-rc scripts are executed. Hopefully, during that 5 mintues, all the ArcSight databases and applications are gracefully stopped. Testing indicated that everything was stopped within 3 minutes, so 5 minutes should be a safe delay.

It's always nice to see a multi-millon dollar product require a customer hack.

skohrs
  • 1,520
  • 11
  • 23