We have a website where we our daily bandwidth use is roughly 10GB of user generated contents.
The software layer is simply Heroku + some addons for DB and monitoring, and we are handling the uploads and data serving with S3 and Cloudfront. The DNS routing is done via Route53.
Now, I thought at first that Cloudfront would be comparable to any other VPN in terms of bandwidth (unclear pricing) but with our rate we are already paying 30$ a month. Compared to any local hosting providers in our country (we really don't need would wide distributed CDN since it's a small country) we would get 300GB a DAY for a fraction of that cost.
To mitigate this issue, we started renting one server and done the following
- Moved all files inside this rented server
- Installed an FTP client inside our heroku app
- Have all the files uploaded to the new "non-cloud" server instead of S3
- Set the URLs for the files to direct to the new files location.
Obviously with this setup, the downside is that we don't get any more CDN, which is almost ok since our traffic is concentrated in one location anyway.
Another downside is the scalability of this: we are obviously at some point going to hit either the bandwidth limit or simply the space limit on the server. So the question is: what are the tools needed to put file storage at scale?
Or, as a side question: I think I either got something very wrong with the setup (in that case please write an anwer or a comment), or why would any sane person choose Cloudfront (or any other CDN?), given the huge cost difference?
