1

I am writing a crawler which regularly inspects a list of news websites for new articles. I have read about different approaches for avoiding unnecessary pages downloads, basically identified 5 header elements that could be useful to determine if the page has changed or not:

  1. HTTP Status
  2. ETAG
  3. Last_modified (to combine with If-Modified-Since request)
  4. Expires
  5. Content-Length.

The excellent FeedParser.org seems to implement some of these approaches.

I am looking for an optimal code in Python (or any similar language) that makes this kind of decision. Keep in mind that header info is always provided by the server.

That could be something like :

def shouldDonwload(url,prev_etag,prev_lastmod,prev_expires, prev_content_length):
    #retrieve the headers, do the magic here and return the decision
    return decision
jaz
  • 175
  • 1
  • 1
  • 7

2 Answers2

2

The only thing you need to check before making the request is Expires. If-Modified-Since is not something the server sends you, but something you send the server.

What you want to do is an HTTP GET with an If-Modified-Since header indicating when you last retrieved the resource. If you get back status code 304 rather than the usual 200, the resource has not been modified since then, and you should use your stored copy (a new copy will not be sent).

Additionally, you should retain the Expires header from the last time you retrieved the document, and not issue the GET at all if your stored copy of the document has not expired.

Translating this into Python is left as an exercise, but it should be straightforward to add an If-Modified-Since header to a request, to store the Expires header from the response, and to check the status code from the response.

kindall
  • 178,883
  • 35
  • 278
  • 309
  • Your answer implies there is no need to care about ETAGs ? And what about content length ? Is not a final way of deciding to download again, when changed, regardless of other params ? – jaz Sep 30 '11 at 12:08
  • If the `Content-Length` or `ETAG` changed, then the document was modified and an `If-Modified-Since` `GET` request will return the document. You could use `ETAG` too in an `If-None-Match` header, but it seems redundant to me to use both, and `ETAG` is optional, so I'd go with `IMS`. – kindall Sep 30 '11 at 13:38
1

You would need to pass in a dict of headers to shouldDownload (or the result of urlopen):

def shouldDownload(url, headers, prev_etag, prev_lastmod, prev_expires,  prev_content_length):
    return (prev_content_length != headers.get("content-length") || prev_lastmod != headers.get("If-Modified-Since") || prev_expires != headers.get("Expires") || prev_etag != headers.get("ETAG"))
    # or the optimistic way:
    # return prev_content_length == headers.get("content-length") and prev_lastmod == headers.get("If-Modified-Since") and prev_expires = headers.get("Expires") and prev_etag = headers.get("ETAG")

Do that when you open the URL:

# my urllib2 is a little fuzzy but I believe `urlopen()` doesn't 
#  read the whole file until `.read()` is called, and you can still 
#  get the headers with `.headers`.  Worst case is you may have to 
#  `read(50)` or so to get them.
s = urllib2.urlopen(MYURL)
try:
    if shouldDownload(s.headers):
        source = s.read()
        # do stuff with source
   else:
        continue
# except HTTPError, etc if you need to  
finally:
    s.close()
chown
  • 51,908
  • 16
  • 134
  • 170