19

My AWS Elastic Beanstalk Environment is stuck in Health: Grey.

enter image description here

My application is working, I can access it fine. However, I am unable to change configuration or deploy new versions because I get a message saying that

Environment named ______ is in an invalid state for this operation. Mustbe Ready.

If I run eb health on my console, I get the following output:

Status: Ready Health Grey

And

ELB State: InService

Is there anything I can try to revive my environment? I have contacted AWS Support, by they are really slow. Another option I can think of is terminating the environment and creating a new one, but I really would prefer to avoid that.

juangesino
  • 444
  • 2
  • 7
  • 10
  • 3
    Every now and then I surface from the matrix and remember that phrases like `Elastic Beanstalk Environment stuck on grey health` are absolutely, unquestionably ridiculous. And then I sink softly beneath the warm waves of technical knowledge and again think of that as a normal thing to say. – Parthian Shot Sep 02 '17 at 00:19

3 Answers3

15

EB can be fairly tough to trouble shoot when you have full access to EB, the instances, the ELB's, etc... never mind trying to proxy this through SO.

I'd do the following:

  • Bring up a new environment under the same application
  • When it comes up green, use the EB application "Swap" functionality to swap the environments

More details on this process is here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.CNAMESwap.html

This performs a DNS switch so you should have no down time. You'll still the old environment running if you want to troublshoot it later with your friendly AWS support staff.

The only negatives are:

  • You'll continue paying for both environment stacks while waiting to troubleshoot the other.
  • The DNS is a little tough as you can't guarantee clients respect the short time-outs EB DNS entries have. They should, but someone may decide to keep using a local cached version. As with anything relying on trusting client-side features, it's a bit out of your control.
  • If you deploy a RDS DB via EB, you can't us the swap as the DB is tied to the environment (NEVER deploy a RDS DB in production EB environment via EB!!!!)
Ray
  • 40,256
  • 21
  • 101
  • 138
  • This is awesome, just one quick question. What happens with my database? Do I have to create a new one or will the new environment use the same DB? Thank you very much! – juangesino Feb 15 '16 at 23:27
  • @JuanGesino Did you use EB to deploy your RDS DB or manually deploy it your self? – Ray Feb 15 '16 at 23:28
  • I used Elastic Beanstalk. – juangesino Feb 15 '16 at 23:30
  • 1
    @JuanGesino Doh!!! That's not good to do for production systems: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.RDS.html It will block your ability to swap as you can't share DB's in different EB environments. Now might be a good time to migrate it to a manually provisioned DB – Ray Feb 15 '16 at 23:32
  • @JuanGesino No problem. you're probably stuck redeploying, sorry. If you're using a single or multi Docker EB app it should be pretty fast/seamless. – Ray Feb 15 '16 at 23:37
  • 3
    Just to throw in my 2 cents...don't EVER deploy RDS with beanstalk. Unless it's all going to be one, very tightly integrated application, I can't think of any reason to do so, certainly not in production. AWS makes it look so convenient and attractive, but you quickly find out how inflexible it is. – Brooks Feb 16 '16 at 00:45
  • BTW, I didn't see any mention of what the logs say about the instance health. You can check the logs in the console, or (if you can't bring them up in the console which happened to be once), you can always SSH into the instance and poke around there. – Brooks Feb 16 '16 at 00:47
  • In this case I think you will loose all configured charts on a Dashboard monitoring related to environment. It's not good – user5629690 Mar 17 '23 at 15:22
5

I know the question's already been answered, but I think the cause of the problem is important, instead of recommending a complete rebuild of OP's environment.

Elastic Beanstalk has 4 different colors - green, yellow, red, and grey. However, each color can mean multiple different things that vary wildly. Here are the potential statuses behind the grey color:

Grey (Suspended) - Your application has had such severe health issues Elastic Beanstalk is no longer monitoring it

Grey (Unknown) - The health agent has not reported enough data on an instance yet

Grey (Pending) - An operation is in progress on an instance within the command timeout (for example bootstrapping the environment)

Notice the incredible disparity between "Pending" and "Suspended". In Pending, it just needs a little more time, or perhaps you can shut down a single resource and have it respawn. In Suspended, all monitoring is shot, and you ought to rebuild the environment ASAP. Big difference in impact to customers during the solution.

Baked into Beanstalk are the vanilla colors. To get the additional statuses, you have to enable Enhanced Monitoring. You can do it in a couple minutes, and the cost is nominal.

To read more about the statuses and common problems with Beanstalk, I'd recommend a blog my colleague wrote: Health Monitoring in AWS Beanstalk

mbarlocker
  • 1,310
  • 10
  • 16
0

A more difficult situation occurs when the environment state is unknown. Even the Abort current operation option doesn't work. To solve this I had to apply these steps:

  • copy the instance id associated with Beanstalk environment
  • find the instance using that instance id in EC2 dashboard
  • stop the EC2 instance
  • start the EC2 instance

The environment should come to the previous state after restarting the instance.

Prem Kumar
  • 11
  • 1
  • 1