4

I'm stumped!

I have a rake task which is cron'd to run every minute.

It's logs in, it finds the JSON that I'm interested in but can take up to 30 runs of the task before any changes in the JSON are noticed in the rake task. During which time I've missed several changes of certain JSON objects.

Seems like there's some caching going on, I've tried to turn off Mechanize caching as shown, just not sure what else I can try now.

Any pointers?

Thanks in advance.

  agent = Mechanize.new # {|a| a.log = Logger.new(STDERR) }
  agent.history.clear
  agent.max_history = 0
  agent.user_agent_alias = 'Mac Safari'
  page = agent.get 'http://website.com'
  form = page.forms.first
  form.email = 'me@home.com'
  form.password = 'mypassword'

  page = agent.submit form
  page = agent.get 'http://website.com/password_protected_page'
  jsonDirty = page.search '//script[@type="application/json"]'

Response from server:

{"server"=>"nginx", "date"=>"Thu, 13 Sep 2012 14:16:43 GMT", "content-type"=>"text/html; charset=utf-8", "connection"=>"close", "vary"=>"Cookie", "content-language"=>"plfplen", "set-cookie"=>"csrftoken=pVDg2SJ4KHqONz2OiEkNK7IbKlnJSQQf; expires=Thu, 12-Sep-2013 14:16:43 GMT; Max-Age=31449600; Path=/, affiliate=; expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/, one-click-join=; expires=Thu,01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/", "expires"=>"Thu, 01 Jan 1970 00:00:01 GMT", "cache-control"=>"no-cache", "content-encoding"=>"gzip", "transfer-encoding"=>"chunked"}
iOSDevil
  • 1,786
  • 3
  • 16
  • 29
  • do you have access to the server logs? If not, would you be able to output the server response headers to the logs? – brycemcd Sep 11 '12 at 19:56
  • Thanks Bryce, I've added the reponse from the server.. – iOSDevil Sep 13 '12 at 14:29
  • Have you verified that the result really changes more frequent? You could try to submit the form using `curl` and compare the javascript. – Roman Sep 15 '12 at 15:12
  • I'll knock up a program to do exactly that, will run it on my laptop rather than the server to be sure. – iOSDevil Sep 16 '12 at 09:02

1 Answers1

1

You could try appending a random query parameter to the URL. Such as:

page = agent.get "http://website.com/password_protected_page?random=#{Time.now.to_i}"
Christian
  • 1,258
  • 10
  • 11