15

My problem is similar to AWS: None of the Instances are sending data but has a slightly different error message.

I have a Rails application running on ElasticBeanstalk, and it appears to be running correctly. Periodically, Enhanced Health Monitoring sends me error messages such as:

Environment health has transitioned from Ok to Degraded. 20.0 % of the requests are failing with HTTP 5xx.

where the percentage varies up to 100%. Even though I've made no changes, a minute later I get a followup message telling me that everything is back to normal:

Environment health has transitioned from Degraded to Ok.

I've downloaded the full logs from ElasticBeanstalk but I don't know exactly where to look (there are around 20 different log files in various directories).

I'm currently using the free AWS tier with the smallest instances of database, server, etc. Could this be the cause? Which of the log files should I be looking in, and what should I be looking for?

Community
  • 1
  • 1
Stephen Lead
  • 1,904
  • 5
  • 23
  • 46

4 Answers4

13

I run rails apps on Elastic Beanstalk and have found it helpful to think about Beanstalk as a computer (in this case an Amazon EC2 instance) running your rails app and a web server (either Passenger or Puma). When you get a 500 error, it could be because your rails app didn't properly deploy–in which case Passenger or Puma will return an error—or your app is deployed properly but encountered an error just like it might on your local machine.

In either case, to diagnose an error, download the full logs from your AWS console (open the correct app environment and then choose Logs > Request Logs > Full logs > Download). Deployment errors are harder to diagnose, but I recommend starting by looking in var-XX/logs/log/eb-activity.log. I suspect your error is coming from your rails app itself, in which case I recommend looking in var-XX/app/support/logs/passenger.log and production.log. To find a 500 error, search for "500 Internal" and then treat the error like you would any other rails error.

apod
  • 578
  • 5
  • 16
  • Thanks for the tip. Passenger.log lists `ActionView::MissingTemplate` but the view named is definitely present (it's the home page). The error message mentions `:formats=>[:xml]` when I'm not using XML - could this be a script/bot trying to access the page? Which might also explain why the problem automatically disappears? – Stephen Lead Nov 02 '15 at 21:51
  • I'll try adding `respond_to :html` to the home page's controller and see if that helps – Stephen Lead Nov 02 '15 at 21:53
  • 1
    There definitely are bots that try to access particular paths on a public site. I'm unsure why, but suspect it's either web crawlers indexing the page or looking for vulnerabilities. Because of this, you'd probably expect to get some 500 errors. I'm unsure why it's causing your environment to not count as healthy. Have you set up a health check link? – apod Nov 02 '15 at 22:16
  • Note: I didn't find it immediately clear from the AWS documentation, but it seems as though the health checker is monitoring web server (nginx, apache) and app server (puma, passenger) logs to get these codes - it's not *just* the ELB health check request/response, as I thought initially. – Jon Burgess Feb 18 '16 at 23:13
3

You can go to the EC2 instance and run the application just like you would run on your local machine and see the logs.

You can ssh into your EC2 instance using the command eb ssh and go to /opt/python/ directory (It will be different for Ruby or other programming languages).

/opt/python/run is the directory where you will find the version of your application which is run from the EC2 instance. Look for the directory venv and app inside run directory.

Note: The above folder structure is for Python but a similar folder structure post deployment can be found for any other programming language. Just look for the standard directory structure for the deployment environment for your programming language.

For Python:

/opt/python: Root of where you application will end up.
/opt/python/current/app: The current application that is hosted in the environment.
/opt/python/on-deck/app: The app is initially put in on-deck and then, after all the deployment is complete, it will be moved to current. If you are getting failures in your container_commands, check out out the on-deck folder and not the current folder.
/opt/python/current/env: All the env variables that eb will set up for you. If you are trying to reproduce an error, you may first need to source /opt/python/current/env to get things set up as they would be when eb deploy is running.
/opt/python/run/venv: The virtual env used by your application; you will also need to run source /opt/python/run/venv/bin/activate if you are trying to reproduce an error.
Akshay Anurag
  • 724
  • 1
  • 8
  • 27
0

I know it is a little late but I wanted to comment the trick I use to find the error, I use to connect via ssh and then, once in the app I try to enter "rails console" It uses to fail, but it shows normally the error you´re making. This little trick saved my life several times. Hope it helps!

Antonio Perez
  • 19
  • 1
  • 2
  • I mean that instead of searching in all log files, sometimes (quite often actually) just running the rails console in the environment (the app in production) shows directly the file in your code where the error is right after trying to launch rails console – Antonio Perez Dec 19 '16 at 12:27
0

To debug the reason this error appears you need to download logs, go to your environment/logs/request logs/full logs. Download logs and go to web.stdout.log. Hopefully now you can debug the problem :)

Oleksandr Nasimov Alex Nas
  • 9
  • 1