6

I have encountered this problem 

None of the Instances are sending data

when deploying a new version of my app on beanstalk, the only diff between the two versions are minor and I'm quite sure it's not the cause. This is what I see:

  • I can not get logs from beanstalk console
  • I can not ssh to EC2 instance of this beanstalk configuration (although the status of that instance is 'running'.
  • Last time I encountered the same issue with a ec2.micro instance, it got solved when I upgrade to ec2.small. What I thought is probably the resource usages make it unresponsive (weird though, since it's only deployment, not even serving any traffic). I don't want to upgrade (again) without really understanding what's happening here.
  • The cpu pattern is like 80% -> 60% -> 20% in during the first 5mins of deployment, then stay at 10% constantly afterwards.

The only way that I can get some server log is to get system log from aws console, and here it is:

Here is the log: https://pastebin.com/PWWjPr3b

Here is what I saw when I issue a ssh:

OpenSSH_7.4p1, LibreSSL 2.5.0
debug1: Reading configuration data /Users/okidogi/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to ec-******.eu-west-2.compute.amazonaws.com 
[35.177.76.128] port 22.
debug1: Connection established.
debug1: identity file aws-eb type 1
debug1: key_load_public: No such file or directory
debug1: identity file aws-eb-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4

Then it's pending there.

Appreciate if someone can help on this.

Feng Li
  • 105
  • 1
  • 8
  • What Elastic Beanstalk "Deployment policy" configuration are you using (All at once - or - Immutable)? If you're using All at once try switching to Immutable, saving the config, letting all the updates apply. Then attempt your deploy again and report back. Health checks are another area to investigate if they're in place on instances or load balancers and your deploy process is longer than a time based failure threshold you could encounter a false positive. – dusthaines Dec 05 '18 at 22:44
  • Thanks @dusthaines ! It was All at once, now it's working after I changed the deployment policy to Immutable.Weird though. Not sure why it didn't work before. – Feng Li Dec 09 '18 at 22:03

1 Answers1

2

Feng, glad to hear that did the trick for you. However, that idea was more of a diagnostic test than a long term solution. Since switching to Immutable worked for you - this would point to the possibility that your build process requires more memory than is available on a (already running) small unit like a t2.micro or t1.micro. By using an Immutable strategy you begin each deploy with a new instance that has more available resources than one that's already consumed with running your app.

This is a problem that is common but hard to diagnose as it presents many different ways depending on platform and framework. You can read more here: https://medium.com/@deanslamajr/an-insufficient-memory-deployment-failure-d9f1cb9b5c0.

My prefered way to address this is through swap memory as I've outlined in an answer to a similar question: AWS EB deploying Node app: failed to run npm install

I would advise trying the .ebextensions strategy mentioned there and also switching back to All at once deployment strategy to test if that truly solves your problem.

dusthaines
  • 1,320
  • 1
  • 11
  • 17