Context
I am using the Docker registry feature in Sonatype Nexus3 (version 3.18.1-01 OSS) to store private Docker images.
A couple of days ago, images with invalid names ended up in the registry, causing multiple reoccurring errors with internal cleanup cron jobs within Nexus3. You can see the faulty repositories names in the v2/_catalog request output below.
$ curl https://registry.example.com/v2/_catalog -i
HTTP/1.1 200 OK
Date: Wed, 11 Mar 2020 12:55:29 GMT
Server: Nexus/3.18.1-01 (OSS)
X-Content-Type-Options: nosniff
Content-Security-Policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation
X-XSS-Protection: 1; mode=block
Docker-Distribution-Api-Version: registry/2.0
Content-Type: application/json
Content-Length: 2941
{
"repositories": [
"MiniProfiler.EF6",
"MiniProfiler.Shared",
...
]
}
At this point, I do not know if these images were pushed in the registry by someone on my team or if they are the result of some unexpected blob store problem - I don't have any way to verify this. Interesting fact: there are NuGet packages named identically in another blob store within the same Nexus instance.
Troubleshooting steps
I have been trying to remove the faulty images from the registry by multiple means, all without success.
Nexus3 GUI
The faulty repositories are not displayed in the Docker registry browser in Nexus' GUI. Neither can they be found by using Nexus' search feature.
Nexus3 embedded OrientDB
I looked for the asset
and component
classes associated with the faulty repositories in OrientDB. Unfortunately, there are no assets or components with such names in Nexus's database.
Docker registry API v2
I also tried to delete them via the Docker registry API V2, but the invalid name format seems to be causing issues when I attempt to do any operation on the faulty repository.
$ curl https://registry.example.com/v2/MiniProfiler.EF6/manifests/latest -i
HTTP/1.1 400 Bad Request
Date: Wed, 11 Mar 2020 12:58:24 GMT
Server: Nexus/3.18.1-01 (OSS)
X-Content-Type-Options: nosniff
Content-Security-Policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation
X-XSS-Protection: 1; mode=block
Docker-Distribution-Api-Version: registry/2.0
Content-Type: application/json
Content-Length: 165
{
"errors": [
{
"code": "NAME_INVALID",
"message": "invalid repository name",
"detail": [
{
"Name": "MiniProfiler.EF6"
},
{
"Reason": "invalid path component: MiniProfiler.EF6"
}
]
}
]
}
Docker CLI
As expected, the Docker CLI doesn't like the invalid name format either.
$ docker pull registry.example.com/MiniProfiler.EF6
invalid reference format: repository name must be lowercase
If I remove the capital letters from the image name, the registry returns a 404 Not Found error.
$ docker pull registry.example.com/miniprofiler.ef6
Using default tag: latest
Error response from daemon: manifest for registry.example.com/miniprofiler.ef6:latest not found: manifest unknown: manifest unknown
Questions
Has anyone ever encountered such problems with Nexus3? Has anyone ever encountered such problems with another Docker registry implementation?
If anyone has any troubleshooting steps to suggest, I'm pretty much open to anything at this point...
I will also be filing a bug with Sonatype to ensure this situation gets good visibility.