6

EB was complaining that my build was timing out, so I ssh'd into an instance and decided to run docker build myself to see what was happening. Every step, even something as simple as a mkdir takes ages to run. Even a WORKDIR stalls for at least a minute or two before executing.

On my local machine these are instant. What is going on?

Kevin Wang
  • 3,290
  • 1
  • 28
  • 40

2 Answers2

8

Same issue here with an Ubuntu machine running on AWS. Turns out the the key to the solution was switching from devicemapper to aufs storage backend.

First, run the following command to figure out which storage backend your currently use:

docker info | grep Storage

If it says devicemapper, you probably found the reason for the slowness.

Here is the prodecure for switching to the aufs backend in Ubuntu, taken from here:

  1. sudo apt-get install -y -q linux-image-extra-$(uname -r)
  2. sudo service docker restart

Note that you will have to rebuild all your existing images / containers, as they will be wiped when you switch to aufs.

urish
  • 8,943
  • 8
  • 54
  • 75
0

Sorry to know you are facing this issue. Elastic Beanstalk environment creation involves creation of lots of resources like autoscaling group, EC2 instances, security groups, Elastic Load Balancer etc. After that software is installed on your beanstalk instances. I am assuming you are only talking about the slowness of software installation (docker build) on beanstalk.

If you just run mkdir that should not be very slow. It should be reasonably fast.

However if you think that docker build overall is running very slow that could be because of IO intensive operations.

One thing you can try is using EBS provisioned IOPs with Elastic Beanstalk. Read more about SSD instances here.

Can you try launching a new environment with SSD instances and see if docker build is still slow? If you can show an example dockerfile that takes a long time to build, I can try it out.

Rohit Banga
  • 18,458
  • 31
  • 113
  • 191
  • Either way a simple `RUN mkdir ~/asd` takes a very long time. This should have nothing to do with EBS boot-related things. I doubt that the FS is really the blocking factor, as I can run `apt-get` and it can fetch things very quickly. – Kevin Wang Sep 21 '14 at 22:56
  • Also the base docker image your dockerfile contains needs to be downloaded (which can easily be several 100MBs). This needs to be downloaded on each EC2 instance for the first time. Do you think that is causing the initial slowness? You can run `docker images` to see what images are currently being downloaded. – Rohit Banga Sep 22 '14 at 04:34
  • These are RUN commands. These happen WAY after the initial image is downloaded. – Kevin Wang Sep 22 '14 at 07:25
  • If its just the RUN command that is slow then it may be something with the docker image ... Is it possible for you to put up some sample Dockerfile that I could use to repro the problem? – Rohit Banga Sep 22 '14 at 07:28
  • It could be... it runs very quickly on my local machine, and my local machine. My instances are medium c1 instances -- pretty similar specs to my laptop. I am on MacOS and building via Boot2Docker, so it should be much more performant than my machine. – Kevin Wang Sep 22 '14 at 07:30
  • 1
    What instance type are you running this on? I started testing then using Docker on Beanstalk recently, and found out much to my dismay that those small `t1.micro` instances could not handle a lot. Once I switched to even t2.small, building the Docker container went from about 10 minutes to 2 minutes. – F.X. Nov 06 '14 at 15:42