Connect to elasticsearch in AWS using key credentials

Question

I'm trying to post a request using curl to my es cluster in AWS using my accessKey and secretKey. I have successfully done this through postman (details here) where you can specify AWS credentials but I would like to make this work with curl. Postman can auto-generate your curl request for you but all I get are errors.

This is the generated curl request along with the response

curl -X GET \
  https://search-00000000000001.eu-west-1.es.amazonaws.com/_cat/indices \
  -H 'Authorization: AWS4-HMAC-SHA256 Credential=11111111111111111111/20181119/eu-west-1/es/aws4_request, SignedHeaders=cache-control;content-type;host;postman-token;x-amz-date, Signature=11111111116401882398f46011f14fdb9d55e012a4fb912706d67c1111111111' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -H 'Host: search-00000000000001.eu-west-1.es.amazonaws.com' \
  -H 'Postman-Token: 00000000-0000-4001-8006-9291e208a000' \
  -H 'X-Amz-Date: 20181119T220000Z' \
  -H 'cache-control: no-cache'

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}%

IDs have been changed to protect the innocent.

I have checked all my keys and region, and like i said this works through postman. Is it possible to access this AWS service using my keys through curl?

It's telling you the issue directly in the error message: it's not that you aren't connecting, it's that your Signature does not match what the service is computing for this message. If this worked from Postman, than it could be that a) you've changed some value being sent; b) Postman is sending some hidden value or header that it didn't export (unlikely); — sofend, Nov 19 '18 at 23:34
Could it be that there is a one time token or something else generated that can only be used once? i know for a fact i didn't change anything. — Damo, Nov 20 '18 at 08:59
I'm afraid I have bad news for you - signing an AWS request to Elastic Search is an extremely demanding, laborious process. Once you get the code right you're fine but it depends on calculating a hash for a very convoluted string - get one character wrong and ... well ... you've seen what happens. Search for "AWS4 signing." This ref gives you a start - https://docs.aws.amazon.com/apigateway/api-reference/signing-requests/ (If I had the time I'd post more details.) Good luck! Adam. — Adam Benson, Nov 20 '18 at 12:45

score 3 · Accepted Answer · answered Nov 21 '18 at 09:24

This is quite a long rabbit hole. Thanks to Adam for the comment that sent me in the correct direction. The link https://docs.aws.amazon.com/apigateway/api-reference/signing-requests/ really helps you understand what you need to do.

I've since found a script that follows the signing requests method outlined above. It runs in bash and whilst it is not written for use with elasticsearch requests it can be used for them.

https://github.com/riboseinc/aws-authenticating-secgroup-scripts many thanks to https://www.ribose.com for putting this on github.

score -2 · Answer 2 · answered Apr 17 '20 at 07:15

If your host contains ':443' remove it and try again. This worked for me.

"My initial problem: If I access it with Postman using the same url, I get the same error, but removing the ‘:443/’, it works fine, so it’s nothing wrong with the key and secret I’m using."

Connect to elasticsearch in AWS using key credentials

2 Answers2