6

Main problem:

  • Application caches URL's from facebook photo CDN
  • The photos expires at some point

My "technical" problem:

  • Facebook CDN "expires" header doesn't seem to be reliable ( or i don't know how to deal with them )

Using CURL to retrieve expiration date:

  • curl -i -X HEAD https://scontent-b.xx.fbcdn.net/hphotos-xap1/v/t1.0-9/q82/p320x320/10458607_4654638300864_4316534570262772059_n.jpg?oh=9d34386036754232b79c2208c1075def&oe=54BE4EE2

  • One minute before it returned: Mon, 05 Jan 2015 01:34:28 GMT

  • Calling it again now returned: Mon, 05 Jan 2015 01:35:27 GMT

  • Both times "Cache-Control" returned the same: Cache-Control: max-age=1209600

So far:

  • It seems like one of the most reliable ways would be to have a background job checking the photos all the time, but that feels a bit "wrong", like "brute forcing".
  • Having a background job would potentially allow expired pictures to be served up to the moment this photo url is "renewed"

My questions are:

  • Should i use the max-age parameter even tough it doesn't seem to change?
  • Is there a reliable way of using facebook's CDN URL?
  • Any other idea of how this should be implemented?
  • < joke >Should facebook API be used to punish badly behaving coders?< /joke >

Possible solutions ?

  • Check facebook for the most recent URL before serving any CDN URL

    ~> would slow down my requests a lot

  • Have a background job renewing the URL and expiration dates

    ~> would potentially have expired photos while the job don't "catch" them

  • Download the photos to my own CDN

    ~> not a good practice i would guess

UPDATE: ~> Perhaps Tinder actually cache user's pictures on their own CDN: https://gist.github.com/rtt/10403467 so seems like facebook is kind of OK with it?

kroe
  • 1,116
  • 3
  • 11
  • 23
  • "would be to have a background job checking the photos all the time, but that feels a bit "wrong", like "brute forcing"." --- and if you think for a second you'll realize that facebook does not know when a user modifies their photo, so they cannot put the exact expiration date. So they just put it as 14 days which they assume is fine. So just do what FB tells you to do with HTTP headers. – zerkms Dec 22 '14 at 03:37
  • Aside from this hypothesis ( user deleting a photo ), do you think i should use the max-age value, or the "Expires" date? – kroe Dec 22 '14 at 03:41
  • Another hardcore issue, is that sometimes it is deleted by users and not deleted from their CDN's - at least used to be that case. – kroe Dec 22 '14 at 03:42
  • 1
    `Expires` == Now + max-age – zerkms Dec 22 '14 at 03:47

1 Answers1

24

Expires means exactly one thing, and it's not what you think it is:

The Expires entity-header field gives the date/time after which the response is considered stale. […]

The presence of an Expires field does not imply that the original resource will change or cease to exist at, before, or after that time.

— RFC 2616 §14.21, emphasis mine

If Facebook's image URLs stop working after some point in time, that's their business. Their HTTP headers don't have to mention it, and in fact, don't.

That being said, I suspect that the oe URL parameter may contain an expiration timestamp. If I interpret 54be4ee2 as a hexadecimal number containing a UNIX timestamp, I get January 20th, 2015, which is almost exactly a month from now. Might that be the value you're looking for?

  • Actually, "Expires" isn't returned for all pictures. So i believe i should use now() + max-age ? – kroe Dec 22 '14 at 05:01
  • 8
    Just to confirm after some tests I have been doing that the parameter oe means when the images will expire in hexadecimal. Good catch @duskwuff – Fran García Jun 27 '16 at 15:07