2

In our project we decided to use presigned urls as basic authentication mechanism.

Trimmed down our setup involves

  • the storage server
  • the api server
  • the client (angular SPA running in the browser)

We use presigned urls for uploading and downloading files from the client directly to the storage server.

Upload flow (simplified):

  • client sends the api: hey I want to upload that
  • api does authorization and validation, does some database stuff and returns a presigned url
  • client uploads directly to the storage server

So far so good. The big problem is the "download" flow.

  • client asks the api: hey, show me a list of what you have
  • api does authorization, validation and returns a json list of objects which also hold the presigned get urls for showing the files (images)
  • the client displays the list of object data and embeds the images downloaded directly from the storage server using the presigned urls

This works great but blows up the browser cache up to multiple GB of RAM.

This happens because the presigned urls generated on mulitple calls are not the same and differ at the authorization part (e.g. holding a new fresh lifetime) on each request. When a user clicks forward and backwards through the paginated list the client will receive different urls and the browser cache treats them as different images.

So far this seems to be a correct behaviour on the browser side (different url equals different image).

So far this seems to be a correct behaviour on the api side (new call will return a new lifetime).

Are there any intended ways how to handle this?

Are the flows themself wrong?

Any ways to solve this beside implementing a centralized presigned url cache when running multiple instances of the api?

May someone could also give advice for meaningful tags I could use.

monty
  • 7,888
  • 16
  • 63
  • 100
  • Could you give each image a static ID and use the image ID, rather than the URL, to track it in the browser cache? If everything is working correctly--as it seems to be--you will need to find a way to 'trick' the browser cache into ignoring the unique URLs in favour of some other identifier. – Josh Harkema Feb 15 '19 at 15:21
  • Hm. We use the presigned get url as src in a standard html img tag (as far as I know, I am the backend guy). The caching is handled by the browsers themself. No idea how or if one could influence the caching behaviour of Chrome or Firefox or any other browser. – monty Feb 15 '19 at 18:09

2 Answers2

1

Every request for a pre-signed resource in your current flow has the browser/client make a new request to S3.

Hence, the benefit of the browser cache isn't enjoyed and can be done without by specifying additional response headers to control caching policy in the client when generating presigned URLs. The Cache-Control response header can be set in the response headers for the pre-signed request to no-cache. 1

A better flow I'll suggest is to have the pre-signed URLs have an expiry time between 5 to 15 mins, and set Cache-Control in the response headers of the response of the pre-signed URL to max-age:<expire-time-in-secs>.

With this new flow, you'll need to ensure that the API server only returns a fresh list of pre-signed URLS after the expiry time by keeping a server side cache as well. You could make gains on the response times from the API server and also avoid serving unnecessary requests from the browser for a cached resource.

Oluwafemi Sule
  • 36,144
  • 1
  • 56
  • 81
  • As I see it right now, I could set the cache control only for the API response (returning a json list of objects, each holding multiple presigned urls). But not for the presigned urls each. – monty Feb 26 '19 at 15:45
1

Another solution is to use presignedurl API from minio-js https://docs.minio.io/docs/javascript-client-api-reference.html#presignedUrl

Please look at https://github.com/minio/minio-js/issues/724 and https://github.com/minio/minio-js/pull/728 for more information.

r1j1m1n1
  • 345
  • 1
  • 4
  • The X-Amz-Date from the referenced issue could be a solution but I don't get it how to use it in C# minio client. – monty Feb 26 '19 at 15:42
  • You can file a feature request at https://github.com/minio/minio-dotnet/issues/new – r1j1m1n1 Feb 27 '19 at 00:24