2

Error
I have a web app with a mass uploader (Plupload) for photos and when I upload say twenty photos, about six (around 30 %) will fail with an Internal Server Error. I have checked the Apache error.log for this domain and it has nothing new (I know I'm looking at the right error.log since older errors did show here).

This only happens on my VPS on Dreamhost (my hosting provider) servers while on my development server it runs silky smooth.

Oh, and things used to work just fine a month ago and then just started to fail. Back then I was using Uploadify and since that used Flash, it was impossible for me to debug where the upload failed.

Files and script
Uploaded files are photos, all about 100 kB big, even though I've successfully uploaded (and still can) 3 MB photos. My .htaccess naturally doesn't change during uploads. On the server side is a PHP script that uses GD2 library to move and resize the photo.

Server state
I have recently upgraded my VPS from 300 to 400 MB of RAM. This thing used to work and I upgraded it just so that memory is ruled out as a reason. Also my memory limit for PHP is at 200 MB, so this should sufice.

I am getting mighty frustrated that Dreamhost does not want to help, stating that "we can not be responsible for an error your code causes" and "We still will not be able to assist you in debugging the issue unfortunately."
It has been a week of sparse "support" while my app doesn't work and my clients are frustrated.

Questions

  1. Is this kind of "You're on your own" support a standard across the industry, i.e. would your host handle this differently?
  2. How exactly can I debug this?
duality_
  • 17,738
  • 23
  • 77
  • 95
  • It says nothing new. I has some old entries, but when I replicate the error, there's nothing new, so you might consider it empty. – duality_ Jun 12 '12 at 09:51
  • Usually, on a shared hosting environment there's a limit to the maximum number of parallel processes which might explain why it's working in a development environment. You might just need to throttle the uploader to process fewer files at once. – Linus Kleen Jun 12 '12 at 10:04
  • Interesting, gonna try that one. *But*, I am on a virtual private server (VPS) so I think this should not be the case (not sure). – duality_ Jun 12 '12 at 10:13
  • Q1: Yes, that seems to be the de facto regarding vps-hosts. Q2: Error 500? Did you turn on error-reporting in php.ini? – Teson Jun 12 '12 at 10:16
  • This VPS is actually managed and they say that "you’ll have around-the-clock (24×7) access to our expert technical support representatives via email, and live chat". I have error_reporting set to E_ALL. – duality_ Jun 12 '12 at 10:37
  • Yes - "php_flag display_errors on" in my .htaccess. – duality_ Jun 12 '12 at 14:31
  • check the directory of the script, i once couldn't find anything in my error log, but then found out it was saving an error log for each folder where the error happens – SAFAD Jun 15 '12 at 07:06
  • If you have display_errors = on for the server side upload target and the uploads are happening in the background, you may not be seeing errors it generates on that side. Do you have log_errors = 1 too? If yes check the log file found in error_log (eg /var/log/apache2/phperror.log) in case it gives any hints – Fanis Hatzidakis Jun 16 '12 at 10:56
  • change log level to "debug" and you will get something more meaningful in logs. (Default is Warn) – Nehal Dattani Jun 18 '12 at 09:11
  • space... if it fails after a cetain storage could be space. – encodes Jun 18 '12 at 14:27

4 Answers4

3

I'm going to assume that you have a standard Apache + PHP setup. One possible configuration is the pre-forked setup; in this case Apache will adapt to system load by forking more children of itself.

With only 400 MB of RAM you're pretty tight, so if you're running 20 processes that each take 200MB (assuming every process handles pretty big files using GD) you're getting into some hot waters with the memory manager.

I would reduce the total number of instances to 2 first to see how this will go; also keep an eye on the memory usage by running top.

Regardless, it might be beneficial for you to run a separate task manager such as Gearman to perform resize tasks so that the upload only has to focus on moving the uploaded file and run the resize task; this way you can greatly reduce the memory required to run your PHP instances.

Ja͢ck
  • 170,779
  • 38
  • 263
  • 309
1

As to your Q1: the simple answer is that you get what you pay for. A 300Mb RAM Dreamhost VPS costs ~$360 per annum. For this you get the VPS service and responses on service failure relating to the provision of the virtual environment. The OS, the software stack and the applications are outside this service scope. Why? This sort of custom knowledge-base support could cost $50-300 per hour. You are being unreasonable and deluding yourself if you expect Dreamhost to provide such services pro-bono. That's what sites like this one do.

So my suggestion is that you suck up that anger and frustration and work out how to help yourself.

As to your Q2. (i) You need to understand where your Apache errors go to; (ii) Ditto any SQL errors if you are using a D/B. (iii) You need to ensure that PHP error logging is enable and verify where the PHP logs are going to. (iv) You need to inspect those logs, and verify that logging is working correctly, by using a small script which generates runtime errors.

You should also consider using enhanced facilities such as php_xdebug to enhance logging levels and introducing application logging.

In my experience systems and functions rarely die silently. However, applications programmers often ignore return statuses, etc. For example in the GD library, imagecopyresized() can fail and it returns a status code to tell the application when it has, but if the application doesn't test this status and act accordingly then it can end up going down bizarre execution paths silently, and just appear to the user (or developer) as "it just stopped working".

My last comment is that you should really consider setting up a private VPS within your development environment which mirrors your Dreamhost production config, and use this for integration, acceptance test and support. This is pretty easy to do and you can mess this and add debug / what if options and then roll back without polluting your production environment. Tools like VMare Appliances and VirtualBox make this easy. See this blog post for a description of how I did this for my hosted service.

TerryE
  • 10,724
  • 5
  • 26
  • 48
0

trying to respond the question 2: if you checked all your code and you didn't see any bug, I thing that the best thing that you can do is to check the version of all the programs running on the server (apache, php, ...), e.g., I remember that I had a problem with a web service it was running on apache and php, the php version was 5.2.8, and after a lot of investigation I found out that that version had a problem parsing xml data.

MARP
  • 581
  • 1
  • 12
  • 19
0

Regarding the first part of the question: Dreamhost do offer a paid support service with "call back". We used this once to get the low down on something. They are very good with general support (better than many hosts IMO) but you can't expect dedicated service, and they must handle a lot of piddling questions. But pay for a call back and, in about 2 minutes on the phone, you can get the answer you want, plus they get their $10 (recurring) for the time. You both win. Just remember to cancel the recurring charges.

Regarding the second part of the question, we had this very same issue with them. Their response (as suggested by Linus in the comments) was that they keep a tally of the CPU use of all processes used by your "user". If that total exceeds a threshold, they will simply kill the process(es) to get the cycles down. No error messages, no warnings, no nothings. Processes can include MySQL, CGI (perl) or PHP. No way to monitor or predict, and we couldn't program round it. Solution... not DreamHost, unfortunately. (webhostingtalk.com will give you loads of host ideas). So we use for some sites, but not for others.

Robbie
  • 17,605
  • 4
  • 35
  • 72