0

Background: I have built a typical website which let's users upload photos and videos (nothing too large, mind). Currently, I host this website on a single DigitalOcean droplet (Nginx, PHP, MySQL, Memcached for sessions/cache) located in London, UK. I have an S3 bucket located in Ireland which I upload the files to, from the server.

The Problem: The problem is for users not located in the UK and Europe the upload speed is slower, sometimes much slower (e.g. from USA or Australia). This leaves them frustrated.

AWS have some pretty cool stuff like S3 Transfer Acceleration and S3 Presigned POSTs (letting users upload directly to a bucket w/o AWS credentials). But the latter isn't very AJAX friendly - you can ask S3 to return an XML response, but as this is cross-domain it's not readable with JS. So there is no way of knowing the key (file) name. AFAIK.

I have also tried uploading directly to an S3 bucket with the AWS JavaScript SDK, but this will mean I have to force my users to re-authenticate, I think (and I want to avoid that).

I was thinking of adding a server in the US for example and separating the services, so two web servers, a session server and a database server. Then use latency based routing with Route 53 to route to the lowest latency web server. But then I have the high latency problem with the session and DB servers in different locations to the web servers. And the sticky sessions problem.

I was wondering if anyone had any suggestions for this, maybe I'm overcomplicating it and there's a simpler solution. Any advice is greatly appreciated!

Tim
  • 31,888
  • 7
  • 52
  • 78
Chris
  • 101
  • 1
  • 3
  • 1
    *"But the latter isn't very AJAX friendly - you can ask S3 to return an XML response, but as this is cross-domain it's not readable with JS."* Well, [cross-domain shouldn't matter](http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html), but POST uploads can simply redirect on success, and the idea of a POST upload is usually that your application chooses the key... so you already would know it. Obvious(?)ly you can't let the user's local filename influence the object key unless you're in an environment where the user is trusted, which is rare. – Michael - sqlbot Aug 12 '16 at 19:19
  • @Michael-sqlbot I tried using this popular jQuery file uploader https://github.com/blueimp/jQuery-File-Upload/wiki/Upload-directly-to-S3 which recommends using the "iframe transport" method to upload. Without the iframe transport, the plugin was sending an `OPTIONS` request to S3. I could actually get the XML response with the iframe method, but couldn't actually access the XML. I think due to this - https://github.com/blueimp/jQuery-File-Upload/wiki/Cross-domain-uploads#cross-site-iframe-transport-uploads. – Chris Aug 12 '16 at 19:46
  • I've found this person who uses the same plugin but doesn't use the iframe method http://pjambet.github.io/blog/direct-upload-to-s3/ although it's a few years old. So I'll try that again. What you said about the key though makes a lot of sense, I'll see if I can create the key when I create the pre-signed post request. In the S3 docs though (and the tutorial above) they speak a lot about using the key of the local file. So that's not great. :-/ https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html – Chris Aug 12 '16 at 19:49
  • *"To use the file name provided by the user, use the ${filename} variable."* They're showing that it's possible to integrate the filename into the key. – Michael - sqlbot Aug 12 '16 at 21:35

1 Answers1

2

You have a few options.

You could set up CloudFront to serve all your content (including pages and uploads) and see if it makes things faster. All traffic, including dynamic, can go over CloudFront, and it's a highly optimized high bandwidth network that in itself can reduce latency quite a bit. That by itself might be enough to make it good enough for your users. Your servers aren't with AWS so this might not work as well, but CloudFront supports any origin. CloudFlare has similar technology, they proxy all your traffic and will have private networks, so that's worth a shot too - they have a free tier.

If you want to add another server in the USA, it's worth just adding a web server. The latency between those points something like 70 - 110ms. I would set up a test server in the USA talking to the London database and see how it works. This will traverse the public internet, so once you're past testing I'd suggest using encryption - either database native or a VPN.

You could put a read replica in the USA, and have write traffic directed to the master database, with reads coming from the read replica. This means most traffic is really fast, but the occasional write will be slightly slower. Since you're not using AWS servers you'd probably have to use the database native replication rather than the simple AWS read replica setup. I'm not sure if read replicas are easy to do with database native or if you have to go to read/write replica.

You could use a dual master database configuration, with replication between them. This could work well if users are unlikely to change existing data in two locations at the same time, or if the schema is such that modifications are rare. Replica lag could be a few seconds, so it could cause some state problems, so I'd probably not try this first.

That's a few ideas, off the top of my head. Give me your thoughts, maybe something else will occur to me. Someone else might have a better solution.

Tim
  • 31,888
  • 7
  • 52
  • 78
  • Thanks Tim, appreciate the response. I should have been more detailed in my post (will update it). I'm using CloudFront for all of the static assets at the moment. The site is a very similar model to Twitter in that there are a lot of reads and writes to the DB, users posting to feeds every minute or so (at peak times) and users reading those feeds. The main problem is the upload time when users upload photos or videos from outside of Europe. It seems a lot slower. – Chris Aug 12 '16 at 18:03
  • Is all your traffic being routed through CloudFront edge locations, or is it just acting as a simple CDN for static objects? If it's the latter look into the former https://aws.amazon.com/cloudfront/dynamic-content/ that way your uploads go to the CloudFront edge location then to your server over the optimised AWS network. – Tim Aug 12 '16 at 18:37