Super-slow Elastic Beanstalk Docker Deployments

Question

I'm deploying a Docker app on Elastic Beanstalk. It works, but takes forever (30 minutes until it's back in the "green" state). On my development environments, it finishes within a few seconds. SSHing into the production instance and tailing the logs (/var/log/eb-activity.log), I see that it hangs on the starred line below (running 03build.sh):

[2016-05-23T13:10:00.430Z] INFO  [3199]  - [Application deployment app-160523_130556@23/StartupStage0/AppDeployPreHook/02loopback-check.sh] : Starting activity...
[2016-05-23T13:10:07.845Z] INFO  [3199]  - [Application deployment app-160523_130556@23/StartupStage0/AppDeployPreHook/02loopback-check.sh] : Completed activity.
*****[2016-05-23T13:10:07.845Z] INFO  [3199]  - [Application deployment app-160523_130556@23/StartupStage0/AppDeployPreHook/03build.sh] : Starting activity... 
[2016-05-23T13:31:58.805Z] INFO  [3199]  - [Application deployment app-160523_130556@23/StartupStage0/AppDeployPreHook/03build.sh] : Completed activity. Result:
  latest: Pulling from srfoster/my-repo
  6714a7dc486b: Pulling fs layer
  5866500c2af9: Pulling fs layer
  ... ETC ...

Based on one of the (not accepted) answers to this similar question...

Why are Docker build commands running so slow in Elastic Beanstalk?

I ran

docker info | grep Storage

and got back

devicemapper

Supposedly, aufs is faster than devicemapper (according to the question linked above).

Buuuut, even if that's true, I wouldn't know how to make Beanstalk's prebuilt images use one versus the other when spinning up.

Any suggestions on this? Or is there some other way to go about fixing the slow deployments?

A few questions: do you deploy using already built Docker image? What registry do use for that - ECR or something else? How many instances of the service you run? — sap1ens, May 23 '16 at 15:57
It's not already built. It uses a base image, though, pulled from a private repo on dockerhub. I see slowness with one EB instance. — Stephen Foster, May 23 '16 at 17:52

score 4 · Accepted Answer · answered May 24 '16 at 00:52

Okay, I tried a deployment using a fully-built docker image to minimize the amount of stuff that had to be done on the instance during deployment. I can confirm that this reduces the deployment time down to about 3 minutes.

Note that (since this is a Rails app) I also had to bake not just the application code but also the results of things like

bundle install

And

rake assets:precompile

I put all this into an image and deploy the fully baked image to (in my case) Dockerhub. What I deployed to Amazon with

eb deploy

was basically just this Dockerfile:

FROM srfoster/my-fully-built-docker-image:latest
EXPOSE 80
ENTRYPOINT /path/to/foreman start -f AWSProcfile

and a barebones Dockerrun.aws.json

{
  "AWSEBDockerrunVersion": "1",
  "Authentication": {
      "Bucket": "my-credentials-bucket",
      "Key": "path/to/my/dockercfg"
  },
  "Ports": [
    {
      "ContainerPort": "80"
    }
  ],
  "Logging": "/var/log/nginx"
}

There was absolutely nothing else in the deployment directory. No application code. Fast deploy. I'm happy.

Super-slow Elastic Beanstalk Docker Deployments

1 Answers1