44

I have a PHP application deployed to Amazon Elastic Beanstalk. But I notice a problem that every time I push my code changes via git aws.push to the Elastic Beanstalk, the application deployed didn't picked up the changes. I checked the events log on my application Beanstalk environment and notice that every time the Beanstalk issues:

Deploying new version to instance(s)

it's always followed by:

The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-d5xxxxx]

The same thing happens when I try to request snapshot logs. The Beanstalk issues:

requestEnvironmentInfo is starting

then after a few minutes it's again followed by:

The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-d5xxxxx].

Cœur
  • 37,241
  • 25
  • 195
  • 267
ardfard
  • 443
  • 1
  • 4
  • 5
  • Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end. – Simon Robb Apr 04 '14 at 00:31
  • I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away? – Chris Carson Apr 24 '14 at 16:25
  • 1
    @Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then. – Simon Robb Apr 26 '14 at 02:16
  • 1
    @SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply. – Chris Carson Apr 28 '14 at 17:04
  • Hi, I'm also experiencing the same problem, did anybody find a solution that doesn't require rebuilding the environment? – jsemig Jun 25 '14 at 08:34
  • 2
    I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond to `git aws.push` if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs. – Chris Carson Jun 28 '14 at 00:55
  • I have an identical issue. Hilariously, the website I am serving is accessible and works like a charm, but EB continues to display the "Still waiting for the following 1 instances to become healthy" message. – hous Sep 02 '15 at 20:13
  • Just to inform that rebuilding is not a solution here. I rebuild my environment but this problem still persists. Due to some reason EB agent on the EC2 is getting died, AWS EB should create a mechanism to autostart it or at least show a more verbose output. Rebooting the instance is the quickest solution. Increasing instance size doesn't look optimal. – vivekyad4v Nov 29 '17 at 06:48

9 Answers9

54

I had this problem a few times. It seems to affect only particular instances. So it can be solved by terminating the EC2 instance (done via the EC2 page on the Management Console). Thereafter, Elastic Beanstalk will detect that there are 0 healthy instances and automatically launch a new one.

If this is a production environment and you have only 1 instance and you want minimal down time

  1. configure minimum instances to 2, and Beanstalk will launch another instance for you.
  2. terminate the problematic instance via EC2 tab, Beanstalk will launch another instance for you because minimum instance is 2
  3. configure minimum instance back to 1, Beanstalk will remove one of your two instances.
chongzixin
  • 1,951
  • 4
  • 28
  • 55
  • 8
    This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens? – Marcus Lind Feb 04 '15 at 08:22
  • 2
    A simple reboot of the instance worked for me once :) – Babu May 05 '15 at 06:26
  • 1
    Stop and resume works for me from EC2. Which creates a new instance for me and move current to new one, and terminate the old one. – Berkay Turancı Jul 02 '15 at 20:37
  • I just rebooted the instance from console and it's worked for me :) – Santosh Dec 24 '15 at 07:10
  • 1
    The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to: `Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping`. The only solution I have found to work is the answer above: reboot (or rebuild). – Logan Pickup Mar 07 '16 at 02:25
  • 1
    Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead. – vivekyad4v Nov 16 '17 at 11:43
  • @Marcus, it kept happening for me too (about 50% of the time). For me, it was purely because of the EC2 instance sizes. The free tier (micro instances) simply isn't enough for even basic apps. Going one step up which **doubled the memory** for me completely fixed the problem. I've not had this reoccur even once in 12 months. It's meant paying ~USD$30/month, but it's definitely worth it. Fyi my use case was a very simple Rails web application. Although if I chose Sinatra/Grape or something else with a smaller memory footprint, I might have gotten away with it for longer. – stwr667 Nov 29 '19 at 05:29
  • I had to force stop mine, wait for it to stop, then start it again. – xinthose Aug 18 '22 at 15:46
8

By default Elastic Beanstalk "throws a timeout exception" after 8 minutes (480 seconds defined in settings) if your commands did not complete in time. You can set an higher time up to 30 minutes (1800 seconds).

{
    "Namespace": "aws:elasticbeanstalk:command",
    "OptionName": "Timeout",
    "Value": "1800"
}

Read here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html

daveoncode
  • 18,900
  • 15
  • 104
  • 159
  • 1
    Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances. – ardfard Apr 04 '14 at 03:33
  • 5
    Where do you put this? – Spoeken Jun 02 '14 at 09:09
  • @Spoeken - in your myapp.config file within your .ebextensions directory – Meistro Aug 08 '14 at 16:18
  • 3600 is the new maximum limit. – Conor Jun 13 '16 at 14:49
  • if you use `eb deploy` from the command line, you can add a `--timeout` parameter followed by the number of minutes you want to set the timeout to be. so for example: `eb deploy --timeout 30 my-prod-app` to set a 30 min timeout for your my-prod-app elasticbeanstalk deploy – Mr. T Sep 03 '20 at 08:10
3

Had the same issue here (single t1.micro instance).

Did solve the problem by rebooting the EC2 instance via the EC2 page on the Management Console (and not from EB page).

mirzik
  • 31
  • 1
3

Beanstalk deployment (and other features like Get Logs) work by sending SQS commands to instances. SQS client is deployed to instances and checks queue about every 20 secs (see /var/log/cfn-hup.log): 2018-05-30 10:42:38,605 [DEBUG] Receiving messages for queue https://sqs.us-east-2.amazonaws.com/124386531466/93b60687a33e19...

If SQS Client crashes or has network problems on t1/t2 instances then it will not be able to receive commands from Beanstalk, and deployment would time out. Rebooting instance restarts SQS Client, and it can receive commands again.

An easier way to fix SQS Client is to restart cfn-hup service:

sudo service cfn-hup restart
Victor Gavrilenko
  • 31
  • 3
1

For me, the problem was my VPC security group rules. According to the docs, you need to allow outbound traffic on port 123 for NTP to work. I had the port closed, so the clock was drifting, and so the EC2's were becoming unresponsive to commands from the Elastic Beanstalk environment, taking forever to deploy (only to time out) failing to get logs, etc.

Thank you @Logan Pickup for the hint in your comment.

0

In the case of deployment, an alternative to shutting down the EC2 instances and waiting for Elastic Beanstalk to react, or messing about with minimum and maximum instances, is to simply perform a Rebuild environment on the target environment.

If a previous deployment failed due to timeout then the new version will still be registered against the environment, but due to the timeout it will not appear to be operational (in my experience the instance appears to still be running the old version).

Rebuilding the environment seems to reset things with the new version being used.

Obviously there's the downside with that of a period of downtime.

Dan Gravell
  • 7,855
  • 7
  • 41
  • 64
  • 2
    It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution. – theBarkman Feb 18 '16 at 18:51
0

I think is the correct way to deal with this. I think the correct way to deal with this is to figure out the cause of the timeout by doing what this answer suggests.

chongzixin's answer is what needs to be done if you need this fixed ASAP before investigating the reason for a timeout.

However, if you do need to increase timeout, see the following:

Add configuration files to your source code in a folder named .ebextensions and deploy it in your application source bundle.

Example:

option_settings:
  "aws:elasticbeanstalk:command":
    Timeout: 2400

*"value" represents the length of time before timeout in seconds.

Reference: https://serverfault.com/a/747800/496353

Pranay Majmundar
  • 803
  • 7
  • 15
0

"Restart App Server(s)" from the "Actions" menu in Elastic Beanstalk management dashboard followed by eb deploy fixes it for me.

Visual cue for the first instruction

Anuj Verma
  • 13
  • 6
0

After two days of checking random issues, I restarted both EC2 instances one after another to make sure there is no downtime. Site worked fine but after a while, website started throwing error 504.

When I checked the http server, nginx was off and "Out of HDD space" was thrown. "Increased the HDD size", elastic beanstalk created new instances and the issue was fixed.

Alvin Chettiar
  • 609
  • 5
  • 14