Unable to import images from public registry

Question

I'm following this guide and when I try to import cert-manager images in my private ACR from command line I receive this error:

(InvalidParameters) Operation registries-561d08e9-81e5-11ed-baec-f834415bade1 failed. Resource /subscriptions/88ea9307-f11d-433e-88c5-7a48cbbfe2f4/resourceGroups/r0b0x/providers/Microsoft.ContainerRegistry/registries/r0b0x1 Error copying blobs. Error copying blobs. Error copying blobs. Error copying blobs. Error copying blobs.

Seems that no one has encountered this error before. Using an azure account you can regenerate the same conditions starting from scratch:

az group create --name sandbox --location eastus
az acr create --resource-group sandbox --name test

# Declare few env variables to use after 
ACR=test
REGISTRY=quay.io
IMAGE=jetstack/cert-manager-controller
TAG=v1.8.0

az acr import --name $ACR --source $REGISTRY/$IMAGE:$TAG --image $IMAGE:$TAG

Do you have any suggestion?

Even if fails, the last command (import) generate something inside my private ACR. If I try to list stored repositories I can see the previously created:

az acr repository list --output table

But if I try to use the image for a deployment or I try to delete it, Azure returns always a resource not found error message. I'm getting crazy with this issue!

What I'm doing wrong?

Beans · Answer 1 · 2022-12-29T07:24:53.303

I too am having this issue. I am following the same Azure MSLearn guide at https://learn.microsoft.com/en-us/azure/aks/ingress-tls?tabs=azure-cli

I worked through this guide in September, and this was not a problem. In fact I ran through it 2 times in September, and this az acr import step did not fail at all.

Note that HELM is not at all involved in this step - this is purely an AZ CLI operation.

I am executing az acr import while logged-in to AZ CLI as the Subscription OWNER, so I have necessary roles to import and delete images.

My experience is that after receiving the error message, I find some or all of the images are in the repository, but something is corrupt. The images cannot be deleted, and they cannot be pulled. Using Azure Portal to attempt to delete the imported repositories results in this dialog:

I am using the same version of AZ CLI as I used in September: 2.38

Here is the import script:

REGISTRY_NAME=myregistry
CERT_MANAGER_REGISTRY=quay.io
CERT_MANAGER_TAG=v1.8.0
CERT_MANAGER_IMAGE_CONTROLLER=jetstack/cert-manager-controller
CERT_MANAGER_IMAGE_WEBHOOK=jetstack/cert-manager-webhook
CERT_MANAGER_IMAGE_CAINJECTOR=jetstack/cert-manager-cainjector

az acr import --name $REGISTRY_NAME --source $CERT_MANAGER_REGISTRY/$CERT_MANAGER_IMAGE_CONTROLLER:$CERT_MANAGER_TAG --image $CERT_MANAGER_IMAGE_CONTROLLER:$CERT_MANAGER_TAG
az acr import --name $REGISTRY_NAME --source $CERT_MANAGER_REGISTRY/$CERT_MANAGER_IMAGE_WEBHOOK:$CERT_MANAGER_TAG --image $CERT_MANAGER_IMAGE_WEBHOOK:$CERT_MANAGER_TAG
az acr import --name $REGISTRY_NAME --source $CERT_MANAGER_REGISTRY/$CERT_MANAGER_IMAGE_CAINJECTOR:$CERT_MANAGER_TAG --image $CERT_MANAGER_IMAGE_CAINJECTOR:$CERT_MANAGER_TAG

running az acr import with --debug we can see some information

urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/xxxxxxxxxx-8551-44e0-ae5b-xxxxxxxx/providers/Microsoft.ContainerRegistry/locations/CENTRALUS/operationResults/registries-xxxxxx-8737-11ed-a5ae-4074e04a4d5d?api-version=2021-08-01-preview HTTP/1.1" 400 315
. . .
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"error":{"code":"InvalidParameters","message":"Operation registries-xxxxxxx-8737-11ed-a5ae-4074e04a4d5d failed. Resource /subscriptions/xxxxxxxxxxxx-8551-44e0-ae5b-xxxxxxxxx/resourceGroups/rg-workflowsaas-nodejs/providers/Microsoft.ContainerRegistry/registries/myregistry Error copying blobs."},"status":"Failed"}
cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception:
cli.azure.cli.core.util: Traceback (most recent call last):
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/polling/base_polling.py", line 517, in run
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/polling/base_polling.py", line 553, in _poll
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/polling/base_polling.py", line 595, in update_status
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/polling/base_polling.py", line 114, in _raise_if_bad_http_status_and_method
azure.core.polling.base_polling.BadStatus: Invalid return status 400 for 'GET' operation

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 231, in invoke
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 663, in execute
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 697, in _run_job
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 333, in __call__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/acr/import.py", line 110, in acr_import
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 1013, in __call__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 1000, in __call__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/polling/_poller.py", line 255, in result
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/polling/_poller.py", line 275, in wait
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/polling/_poller.py", line 192, in _start
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/polling/base_polling.py", line 521, in run
azure.core.exceptions.HttpResponseError: (InvalidParameters) Operation registries-xxxxxxx-8737-11ed-a5ae-4074e04a4d5d failed. Resource /subscriptions/xxxxxx-8551-44e0-ae5b-xxxxxx/resourceGroups/rg-workflowsaas-nodejs/providers/Microsoft.ContainerRegistry/registries/myregistryError copying blobs.
Code: InvalidParameters
Message: Operation registries-xxxxxxx-8737-11ed-a5ae-4074e04a4d5d failed. Resource /subscriptions/xxxxxxxx-8551-44e0-ae5b-xxxxxx/resourceGroups/rg-workflowsaas-nodejs/providers/Microsoft.ContainerRegistry/registries/myregistryError copying blobs.

I found this Q&A from MSLearn which attributes this to quay.io not supporting range blob operations which are used by acr import. It goes on to suggest manually downloading and then pushing into ACR.

But I will repeat, this worked in September, using the same versions of clients. Seems like something broke recently? Anyway this seems to be the answer.

https://learn.microsoft.com/en-us/answers/questions/1136080/unable-to-import-image-to-container-registry.html

Running into the same issue, I'm doing the docker pull/tag/push manually. — Marouane Gazanayi, Jan 03 '23 at 17:30

Beans · Answer 2 · 2023-02-05T06:12:27.940

There was an answer for this posted in learn.microsoft.com, acknowledging this is a recently evolved issue - perhaps quay recently stopped supporting range operations - and suggesting to import the packages into your own docker registry and deploying from there. Prior to Summer/Fall of 2022, az acr import was able to import from quay.

It's not an issue with Helm or wth permissions; the issue is that az acr import is not compatible with quay's Api which does not support range operations that az acr import uses.

https://learn.microsoft.com/en-us/answers/questions/1136080/unable-to-import-image-to-container-registry

Swarna Anipindi · Answer 3 · 2023-01-04T03:45:40.643

-2

ACR Import needs authentication implicitly, provide username and password values as mentioned below.

Enable Admin User at Access Key level
Login ACR[destination]

az acr login -n <container registry name> --expose-token

enter image description here

Commands to copy the image.

$source = “Source Container”  
$imageTag=“Image”  
$destination="Destination"
$username= “Src username”  
$password= "passw"
az acr login -n --expose-token  
az acr import --name  destination --source "destination−−source"[Source](http://source.azureacr.io)./$imageTag" --username $username --password $password

Note: Need to provide the username and password values implecitly to acr import command along with image tag. Grab them from step1 screen. refer

edited Jan 04 '23 at 03:45

answered Dec 23 '22 at 06:14

Swarna Anipindi

792
2
9

1

Sorry but my problem is not related to an authentication factor. I'm able to import other images but not these one mentioned. I think there is some kind of restriction related to the size of the image and the SKU of used ACR. – Roberto Conte Rosito Dec 27 '22 at 10:43
Issue was caused because of Helm dependency on ACR. As mentioned in above doc reference we should have the latest release of Helm and have access to the ingress-nginx and jetstack Helm repositories. refer this https://learn.microsoft.com/en-us/azure/container-registry/container-registry-helm-repos. – Swarna Anipindi Dec 27 '22 at 11:59
I do not believe this is related to HELM. The command is purely an AZ CLI command, ie: `az acr import`, there is not use of HELM at all. I have this same error happening, I am authenticated as the subscription Owner and I can import other images from mcr.microsoft.com. This was working in September, same az cli version and container SKU. I was following the same Azure AKS guide. – Beans Dec 29 '22 at 04:58
docker pull quay.io/jetstack/cert-manager-controller:v1.8.0 succeeds. But when imported via az acr import with --debug, one can see there is a failure. – Beans Dec 29 '22 at 05:04
Take a look at the answer, quay.io doesn't support range blob operations – Marouane Gazanayi Jan 03 '23 at 17:31

Unable to import images from public registry

3 Answers3