1

I have installed Airflow 2.0 using Docker on AWS and trying to pull my own private docker image GitLab container registry using DockerOperator but getting permission denied.

How I can pull my docker image from GitLab and How I can create a connection for the Docker image GitLab registry?

Dag file link Docker yml file link

My dag Code:

java_test_scheduler = DockerOperator(
    task_id='Java-hello-world-test-scheduler',
    image='registry.gitlab.com/mobinalhassan/jamay_aeronova:latest',
    auto_remove=True,
    force_pull=True,
    dag=dag
)

I also tried by adding docker volums:

- /var/run/docker.sock:/var/run/docker.sock
- $HOME/.ssh:/root/.ssh:ro
- $HOME/.docker:/root/.docker

Error:

sock.connect(self.unix_socket) PermissionError: [Errno 13] Permission denied
Mobin Al Hassan
  • 954
  • 11
  • 22

2 Answers2

0

Your dag is missing dag_conn_id definition when pulling from private docker registry:

If a login to a private registry is required prior to pulling the image, a Docker connection needs to be configured in Airflow and the connection ID be provided with the parameter docker_conn_id.

https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/_api/airflow/providers/docker/operators/docker/index.html

Ladislav Zigo
  • 484
  • 4
  • 9
  • I have tried this but failed, There is no documentation on how to create a connection link... – Mobin Al Hassan May 03 '21 at 11:16
  • This is the link I have created...docker://gitlab+68hzvkEtonsazVoxNHsy:mobin@registry.gitlab.com but not working, Can you explain that.. – Mobin Al Hassan May 03 '21 at 11:18
  • Are you using airflow connections? https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html then use your conn id (for example 'gitlab_conn_id') in dockeroperator initialization code as parameter `docker_conn_id='gitlab_conn_id' ` – Ladislav Zigo May 03 '21 at 16:12
  • Yes! actually, I did the whole thing, setting up the connection and giving Id in DockerOperator as you have mentioned... – Mobin Al Hassan May 04 '21 at 05:51
0
  1. I have tried without docker you can check this answer for details on that
  2. I wanted to keep least dependency on airflow and hence didn't create the connection from airflow whose details are mentioned in the above referenced answer link
  3. As far as your dag link is concerned, I had tried similar approach but could not make it work with image='registry.gitlab.com/mobinalhassan/jamay_aeronova:latest', however it did work out when I removed the :latest and use it like image='registry.gitlab.com/mobinalhassan/jamay_aeronova',, try this out and let me know if it works
  4. For full details on pulling private docker image, I have jotted down a step by step tutorial which you can refer at - this link
imsheth
  • 31
  • 2
  • 18
  • 36