1

Our production server is running Apache v2.2.4 on CentOS5.2. Mono v1.2.4 is integrated within Apache.

Recently, we faced a problem in our production server. From Apache's access_log, I found a HTTP 500 internal server error for one of the HTTP request and all subsequent HTTP requests also failed but with HTTP 503 service unavailable error. From thereafter, none of the requests were successful. Also, only later some time, we realized that our application was not working because of this error and then we restarted Apache service.

My questions are,

  1. in this kind of situation, how do I automatically restart Apache service when HTTP 503 error is encountered? Is there any Apache directive available to set?
  2. in general, what would cause a HTTP 503 error in Apache?

NOTE: Mono helps in running applications developed in .NET on a Linux-based OS.

EDIT: I agree on finding the root cause of this problem. In fact, we've been analyzing that too. Till we resolve it, am finding whether this could be restarted immediately on its own without having any downtime/service disruption for application users.

Gnanam
  • 1,459
  • 13
  • 26
  • 32
  • So, overall from the 5 answers given below, it appears that there is no default built-in Apache directive available to restart service on its own whenever Http 503 error is encountered. Solution for this situation is of 2 kind: 1) Setting up monitoring tools like `monit` or `swatch` and then restarting Apache service based on error status. 2) Shell script that monitors/reads Apache's error/status log at defined interval and restarts whenever an error status is encountered. – Gnanam Feb 25 '11 at 13:19

6 Answers6

3

you should really fix the problem, but you could run something like this for a temp solution:

#!/bin/sh                                                                                                            
HTTP_STATUS=`lynx -head -source http://mysite.com |head -1 |awk '{print $2}'`                             
if [ "$HTTP_STATUS" = "503" ]                                                                                       
then                                                                                                                 
        /etc/init.d/http restart                                                                                   
fi                                                                                                                   
aspitzer
  • 977
  • 5
  • 14
2

503 errors are normally related to errors on an application being served or a configuration error. It should be on apache's error log (if you don't have an error log, enable it on the configuration with the ErrorLog directive and control what level of error logging you want with LogLevel).

You can use monit to check a page and see if the HTTP code from a determined page is ok, and what to do if not (like, restarting apache).

coredump
  • 12,713
  • 2
  • 36
  • 56
  • error log is already enabled in Apache. But how do I instruct Apache to restart on its own automatically whenever 503 error is encountered? – Gnanam Feb 18 '11 at 12:16
  • Using monit is the way I suggested to do it. – coredump Feb 18 '11 at 13:02
  • But I agree with @katriel on that you should try to find the root case of the problem, restarting apache on a error is just a temporary solution that will bite you on the future. – coredump Feb 18 '11 at 13:03
  • +1 for letting me know the existence of yet another free open source utility for managing and monitoring, processes, files, directories and filesystems on a UNIX system - `Monit` – Gnanam Feb 18 '11 at 13:14
  • Monit is king. I don't use it's web interface, just the daemon to monitor stuff that can't go down. – coredump Feb 18 '11 at 16:10
2

I don't think you can, or should. You're much better leaving the server as-is and trying to find the root of the problem. IMHO, restarting on error is a bad policy in general.

If you still insist, try looking at something like SWATCH to monitor your apache logs and act on a 503 error.

katriel
  • 4,477
  • 23
  • 20
1

You could have something like the following in your httpd.conf:

ErrorDocument 500 /cgi-bin/handle-500-error.pl

That would call a script whenever an error is encountered, and set this up to restart the server.

HopelessN00b
  • 53,795
  • 33
  • 135
  • 209
Ivan
  • 11
  • 1
0

I've encountered a 503 error when something I'm reverse proxying behind Apache is down. Restarting the service that is reverse proxied fixes it, but this error is expected if I've shut down that service.

Perhaps some extension is attempting to contact another service that isn't running, or is timing out.

LawrenceC
  • 1,202
  • 7
  • 14
  • I too agree with you on finding the root cause of this problem. In fact, we've been analyzing that too. Till we resolve it, am finding whether this could be restarted immediately on its own without having any downtime/service disruption for application users. – Gnanam Feb 18 '11 at 13:10
  • Sorry, I was about to add my above comment against @katriel and added here mistakenly. – Gnanam Feb 21 '11 at 10:41
0

This is what I understand you want:

You want something to monitor the log and INSTANTLY restart apache, if a specific error occurs. You do not want to probe every minute, just watch log.

You can achieve that, but it might give you a performance issue, although this won't cost you much server power.

However! This is generally not a good idea, as some person could find out what you were looking for in the log file and then just keep on making 404 errors with that string at the end of the URL. But:

Edit the config file and pipe the errorlog to a script, such as:

ErrorLog "| /bin/bash /scripts/logMonitor.sh"

Then create that script and it could be something like this:

#!/bin/bash

while read input; do

  if [[ $input == *someError* ]]; then
    echo "--- ERROR REGISTERED - RESTARTING APACHE ---" >> /tmp/err.log
    /etc/init.d/apache2 restart
  fi    

  # Always log what is thrown at us.
  echo "$input" >> /tmp/err.log

done

This will do what you ask for, but it is ugly and not recommended.

Frands Hansen
  • 4,657
  • 1
  • 17
  • 29