10

Since dockerhub has started limiting downloads for non-paid accounts, I am frequently getting this error

ERROR: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (executor_docker.go:188:15s)

https://docs.docker.com/docker-hub/download-rate-limit/

I have an artifactory server, how can I setup docker to cache/mirror from artifactory first?

spuder
  • 17,437
  • 19
  • 87
  • 153

3 Answers3

8

First setup an artifactory remote that points to docker hub at https://registry-1.docker.io

enter image description here

Then reconfigure all docker agents to use the registry_mirror. There are multiple ways to do this, but adding --registry-mirror to the docker start up options is most likely the easiest. See the docker documentation for more information

https://docs.docker.com/registry/recipes/mirror/

Method 1

Add --registry-mirror to the OPTIONS variable in /etc/default/docker

cat /etc/default/docker
OPTIONS=" -H unix:///var/run/docker.sock --ip-forward=true --iptables=true --ip-masq=true --registry-mirror=https://docker.artifactory.example.com -G docker"

Method 2

Edit /etc/docker/registry/config.yml

proxy:
  remoteurl: https://registry-1.docker.io
  username: [username]
  password: [password]

Method 3

If using puppet the config looks like this

  class { '::docker':
    use_upstream_package_source => false,
    manage_package              => false,
    registry_mirror             => 'https://docker.artifactory.example.com',
  }
spuder
  • 17,437
  • 19
  • 87
  • 153
5

Steps to avoid running into Dockerhub rate limits:

  1. Sign up for a Dockerhub free account if you do not already have one. Dockerhub enables credential pulls of up to 200 per 6 hours vs 100 per 6 hours for anonymous pulls.

  2. Use Artifactory as a cache between Dockerhub by setting up a remote repo to Dockerhub and a local repo to push and pull images that are not on dockerhub.

  3. Avoid using Dockerhub for personal images. Only pull official images as necessary.

  4. Setup your docker clients to always pull through Artifactory by using the docker repo path of the virtual repo ex:

    docker.artifactory.example.com/docker-virtual/myimage:1.0.0

  5. Pull official images the same way by using the path w/ the virtual repo in it.

    docker.artifactory.example.com/docker-virtual/ubuntu:latest

  6. Monitor your Dockerhub rate limits through the use of analytics JFrog has provided integrations into Splunk, Elastic, and Prometheus to monitor your rolling 6 hour window of dockerhub pull requests and cache hit ratio.

John Peterson
  • 354
  • 1
  • 4
4

You can also get some information in an official blog JFrog released related to this.

See detailed instructions from @john-peterson in this page.

Eldad Assis
  • 10,464
  • 11
  • 52
  • 78
  • 1
    This blog was also published for monitoring this setting through analytics: https://jfrog.com/blog/keep-watch-on-docker-hub-pulls-with-jfrog-log-analytics/ – John Peterson Nov 04 '20 at 22:27
  • The blog posts don't explain how to use artifactory as a docker hub mirror, they just claim that it is possible. – artm Jun 02 '21 at 06:42