ceph s3 radosgw 403

Question

I have ceph and radosgw installed, which are working pretty well with s3cmd. I decided to setup docker registry version 2 and use s3 as storage. I'm starting it like:

docker run --name registry -d -p 5000:5000 \
    -e "REGISTRY_STORAGE=s3" \
    -e "REGISTRY_STORAGE_S3_REGION=eu" \
    -e "REGISTRY_STORAGE_S3_REGIONENDPOINT=http://s3.int.*.com" \
    -e "REGISTRY_STORAGE_S3_BUCKET=registry" \
    -e "REGISTRY_STORAGE_S3_ACCESSKEY=aaa" \
    -e "REGISTRY_STORAGE_S3_SECRETKEY=xxx" \
    -e "REGISTRY_STORAGE_S3_ROOTDIRECTORY=/" \
    registry:2

and when I'm trying to push something, I'm getting

The push refers to a repository [localhost:5000/myfirstimage]
9a39129ae0ac: Pushing [==================================================>] 4.608 kB
98305c1a8f5e: Pushing 9.728 kB
0185b3091e8e: Pushing [==================================================>] 15.87 kB
ea9f151abb7e: Pushing [==>                                                ] 5.379 MB/124.8 MB
error parsing HTTP 403 response body: unexpected end of JSON input: ""

In radosgw logs

2016-07-28 17:13:05.426328 6ea3a7347700 20 RGWWQ:
2016-07-28 17:13:05.426330 6ea3a7347700 20 req: 0x6ea43801b760
2016-07-28 17:13:05.426338 6ea3a7347700 10 allocated request req=0x6ea43801da60
2016-07-28 17:13:05.426351 6ea39f4e4700 20 dequeued request req=0x6ea43801b760
2016-07-28 17:13:05.426362 6ea39f4e4700 20 RGWWQ: empty
2016-07-28 17:13:05.426419 6ea39f4e4700 20 CONTEXT_DOCUMENT_ROOT=/var/www/html
2016-07-28 17:13:05.426421 6ea39f4e4700 20 CONTEXT_PREFIX=
2016-07-28 17:13:05.426423 6ea39f4e4700 20 DOCUMENT_ROOT=/var/www/html
2016-07-28 17:13:05.426424 6ea39f4e4700 20 FCGI_ROLE=RESPONDER
2016-07-28 17:13:05.426425 6ea39f4e4700 20 GATEWAY_INTERFACE=CGI/1.1
2016-07-28 17:13:05.426426 6ea39f4e4700 20 HTTP_CONNECTION=close
2016-07-28 17:13:05.426428 6ea39f4e4700 20 HTTP_HOST=s3.int.mulbabar.com
2016-07-28 17:13:05.426431 6ea39f4e4700 20 HTTP_REFERER=http://localhost:5000/v2/myfirstimage/blobs/sha256:670a583e1b5043337f8c73dcad9ff019f2ac0971475456fd4fdf296f2c5b939f
2016-07-28 17:13:05.426432 6ea39f4e4700 20 HTTP_USER_AGENT=docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/4.4.0-31-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \(linux\))
2016-07-28 17:13:05.426433 6ea39f4e4700 20 HTTP_X_FORWARDED_FOR=10.0.0.248
2016-07-28 17:13:05.426434 6ea39f4e4700 20 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2016-07-28 17:13:05.426436 6ea39f4e4700 20 QUERY_STRING=X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=V26NKHZPDQ73YAFOERYW%2F20160728%2Feu%2Fs3%2Faws4_request&X-Amz-Date=20160728T151305Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=3d255d8f0ee62b2fa0b1cf9f84fc52a7e2b187134dbc604e6be80dfec71164e3
2016-07-28 17:13:05.426439 6ea39f4e4700 20 REMOTE_ADDR=10.0.0.251
2016-07-28 17:13:05.426441 6ea39f4e4700 20 REMOTE_PORT=39814
2016-07-28 17:13:05.426442 6ea39f4e4700 20 REQUEST_METHOD=HEAD
2016-07-28 17:13:05.426443 6ea39f4e4700 20 REQUEST_SCHEME=http
2016-07-28 17:13:05.426445 6ea39f4e4700 20 REQUEST_URI=/registry/docker/registry/v2/blobs/sha256/67/670a583e1b5043337f8c73dcad9ff019f2ac0971475456fd4fdf296f2c5b939f/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=V26NKHZPDQ73YAFOERYW%2F20160728%2Feu%2Fs3%2Faws4_request&X-Amz-Date=20160728T151305Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=3d255d8f0ee62b2fa0b1cf9f84fc52a7e2b187134dbc604e6be80dfec71164e3
2016-07-28 17:13:05.426448 6ea39f4e4700 20 SCRIPT_FILENAME=/var/www/html/s3gw.fcgi
2016-07-28 17:13:05.426449 6ea39f4e4700 20 SCRIPT_NAME=/registry/docker/registry/v2/blobs/sha256/67/670a583e1b5043337f8c73dcad9ff019f2ac0971475456fd4fdf296f2c5b939f/data
2016-07-28 17:13:05.426450 6ea39f4e4700 20 SCRIPT_URI=http://s3.int.mulbabar.com:8090/registry/docker/registry/v2/blobs/sha256/67/670a583e1b5043337f8c73dcad9ff019f2ac0971475456fd4fdf296f2c5b939f/data
2016-07-28 17:13:05.426451 6ea39f4e4700 20 SCRIPT_URL=/registry/docker/registry/v2/blobs/sha256/67/670a583e1b5043337f8c73dcad9ff019f2ac0971475456fd4fdf296f2c5b939f/data
2016-07-28 17:13:05.426452 6ea39f4e4700 20 SERVER_ADDR=10.0.0.1
2016-07-28 17:13:05.426454 6ea39f4e4700 20 SERVER_ADMIN=[no address given]
2016-07-28 17:13:05.426455 6ea39f4e4700 20 SERVER_NAME=s3.int.*.com
2016-07-28 17:13:05.426456 6ea39f4e4700 20 SERVER_PORT=8090
2016-07-28 17:13:05.426457 6ea39f4e4700 20 SERVER_PROTOCOL=HTTP/1.1
2016-07-28 17:13:05.426458 6ea39f4e4700 20 SERVER_SIGNATURE=
2016-07-28 17:13:05.426459 6ea39f4e4700 20 SERVER_SOFTWARE=Apache/2.4.10 (Debian)
2016-07-28 17:13:05.426461 6ea39f4e4700  1 ====== starting new request req=0x6ea43801b760 =====
2016-07-28 17:13:05.426483 6ea39f4e4700  2 req 83:0.000021::HEAD /registry/docker/registry/v2/blobs/sha256/67/670a583e1b5043337f8c73dcad9ff019f2ac0971475456fd4fdf296f2c5b939f/data::initializing for trans_id = tx000000000000000000053-00579a2101-49cca-default
2016-07-28 17:13:05.426490 6ea39f4e4700 10 host=s3.int.*.com
2016-07-28 17:13:05.426495 6ea39f4e4700 20 subdomain= domain=s3.int.*.com in_hosted_domain=1 in_hosted_domain_s3website=0
2016-07-28 17:13:05.426552 6ea39f4e4700 20 get_handler handler=22RGWHandler_REST_Obj_S3
2016-07-28 17:13:05.426557 6ea39f4e4700 10 handler=22RGWHandler_REST_Obj_S3
2016-07-28 17:13:05.426560 6ea39f4e4700  2 req 83:0.000098:s3:HEAD /registry/docker/registry/v2/blobs/sha256/67/670a583e1b5043337f8c73dcad9ff019f2ac0971475456fd4fdf296f2c5b939f/data::getting op 3
2016-07-28 17:13:05.426565 6ea39f4e4700 10 op=21RGWGetObj_ObjStore_S3
2016-07-28 17:13:05.426567 6ea39f4e4700  2 req 83:0.000106:s3:HEAD /registry/docker/registry/v2/blobs/sha256/67/670a583e1b5043337f8c73dcad9ff019f2ac0971475456fd4fdf296f2c5b939f/data:get_obj:authorizing
2016-07-28 17:13:05.426616 6ea39f4e4700 10 NOTICE: now = 1469718785, now_req = 1469715185, exp = 1200
2016-07-28 17:13:05.426619 6ea39f4e4700 10 failed to authorize request
2016-07-28 17:13:05.426621 6ea39f4e4700 20 handler->ERRORHANDLER: err_no=-1 new_err_no=-1
2016-07-28 17:13:05.426656 6ea39f4e4700  2 req 83:0.000195:s3:HEAD /registry/docker/registry/v2/blobs/sha256/67/670a583e1b5043337f8c73dcad9ff019f2ac0971475456fd4fdf296f2c5b939f/data:get_obj:op status=0
2016-07-28 17:13:05.426660 6ea39f4e4700  2 req 83:0.000198:s3:HEAD /registry/docker/registry/v2/blobs/sha256/67/670a583e1b5043337f8c73dcad9ff019f2ac0971475456fd4fdf296f2c5b939f/data:get_obj:http status=403
2016-07-28 17:13:05.426664 6ea39f4e4700  1 ====== req done req=0x6ea43801b760 op status=0 http_status=403 ======

I have found that Amazon supports "Authenticating Requests: Using Query Parameters (AWS Signature Version 4)" http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html

So, my questions: 1. Is there any way to tell radosgw to authenticate requests using query qarameters? 2. Is there any way to tell registry not to use such authentication? 3. Is there any way to use registry with radosgw? (maybe swift?)

Thanks in advance, Rafael

score 0 · Answer 1 · answered Aug 21 '17 at 05:46

It's not clear what version of Ceph you have deployed, but based on the RGW debug output you provided, I think you have an older version that didn't have AWS v4 signature support. AWS v4 signatures were first supported in Jewel, released in April 2016, but still contained bugs with up to now even.

Specifically, depending how Docker handles the authentication via AWSv4 query string, you might hit this bug: http://tracker.ceph.com/issues/20992

There is a backport waiting for merging to Jewel: https://github.com/ceph/ceph/pull/17009 And if you do want to use it, make sure to add the configuration setting to your rgw's ceph.conf.

ceph s3 radosgw 403

1 Answers1