0

We are using gunicorn + gevent on heroku, with our own hosted redis, and we are losing data at an average of 10 jobs for every 2000 jobs we put in. We have multiple points of redundancy and tracking, hence we are able to detect that the jobs did not go through.

Anyone has any idea what might have lead to the disappearing of jobs?

Thanks!

laurenceputra
  • 317
  • 2
  • 5
  • 16
  • what is the question? – usethedeathstar Jul 16 '13 at 07:38
  • oops sorry, for some reason the question was truncated. it has been updated now. – laurenceputra Jul 16 '13 at 08:09
  • Can you explain what parts you are tracking and where you are seeing the jobs start to disappear? Is it a particular place you see the job dropping or just that work in the end isn't finished? – Philip Cristiano Jul 17 '13 at 10:36
  • so i monitored it live, when it got the job, it will print on stdout, and send it over to redis through rq, and i was running monitor on redis. some jobs do get though, but a majority doesn't even appear on redis, and throws no error on python too – laurenceputra Jul 19 '13 at 09:35
  • I haven't heard of Redis accepting writes that wouldn't show up in `monitor`. I'd recommend coming up with as simple of a script as possible to demonstrate this. My best guess is that some greenlets are getting blocked for some reason or dying unexpectedly but without code that's all I can suggest. – Philip Cristiano Jul 22 '13 at 13:26
  • ah well, moved to mongo and lost nothing so far. thanks for the help :) – laurenceputra Jul 25 '13 at 06:07
  • @laurenceputra thats how you solve problems sir :P – Tommaso Barbugli Aug 23 '13 at 08:00

0 Answers0