0

Is there any way to tell S3 to wait for an object before responding? I want to do this directly against the S3 endpoint via an HTTP(S) request.

I understand this function exists in the PHP SDK http://docs.aws.amazon.com/aws-sdk-php/v2/guide/feature-waiters.html#

But can you apply this wait functionality when downloading a resource directly from the S3 endpoint? (without an SDK)?

Paul Fryer
  • 9,268
  • 14
  • 61
  • 93

1 Answers1

1

The answer is yes, but not really.

Yes, because everything the SDK can do can be done with your own code, because the SDK uses the same documented public APIs to access the services... the SDKs have no privileged interface... but, then again, not really, because the waiter logic is pretty much an illusion, a convenient fiction. It's not waiting for the service in a "tell me when you're ready" sense... it's really just polling the service to check the state of the resource in question. They're just a convenience so you don't have to write your own polling retry logic.

So, sure you can... but to do it, you just have to keep asking until you get the response you expect... which I assume is not the answer you were hoping for.

When you say "wait for an object," though, a new object (not an overwrite) should be available as soon as the PUT returns success. Overwrites and deletions are eventually-consistent, but new object consistency is immediate.

One exception to this is in the us-east-1 region of S3 where immediate consistency is only officially provided if you use the s3-external-1.amazonaws.com or s3.dualstack.us-east-1.amazonaws.com endpoints, not the s3.amazonaws.com endpoint.

Michael - sqlbot
  • 169,571
  • 25
  • 353
  • 427
  • Michael thanks for your response. You are right I'm no talking about polling via an SDK. I'm actually talking about asking S3 to poll for me and keep the HTTP request open until the object actually manifests (comes into existence) on the server. I've got a browser that needs to load resources from a server, but not all the resources are available at the moment it wants them. I'd like to be able to point my browser at these locations that will be fulfilled within like 10 seconds. So the objects are being created asynchronously and I want to wait for them synchronously, if that makes any sense. – Paul Fryer Oct 19 '16 at 23:31
  • So I think the answer to this question is; No, S3 doesn't have this capability inherently from the server side. If you want it add a feature request with the AWS team. – Paul Fryer Oct 19 '16 at 23:34
  • Correct, S3 does not have this native capability. I am not affiliated with AWS, so I can't speak to its viability as a feature request, though it seems somewhat unlikely. The capability could be emulated in Javascript or with a proxy server that retries the requests or integrates with [S3 event notifications](https://aws.amazon.com/blogs/aws/s3-event-notification/). – Michael - sqlbot Oct 20 '16 at 02:00
  • Yes I completely understand you can do polling client side or with a proxy. For my specific use case I don't have the ability to use a client side SDK and I don't want to implement a proxy if I don't have to (for cost reasons). In any case I think we have our answer. Thanks for the discussion. – Paul Fryer Oct 20 '16 at 14:35