1

This script is executed by job scheduler Azkaban every day and have been working fine for more than a week now. But yesterday it failed unexpectedly. No exception was thrown, no fatal error and no error log entry. I am inserting a count of records processed to database from destruct function that worked fine but shows that only 241 of 7060 records where processed.

Where I am at now:

  1. If district was called that means no uncaught exception, no fatal error.
  2. 241 records processed to script was working fine.
  3. no error in php error log, syslog, logs of azkaban
  4. script stopped after 50 seconds. normally it takes 30+ minutes. so not timeout.
  5. it is unlikely to exceed Memory limit. only fetching and processing one record at a time.

Is it possible OS or azkaban or some other process stopped my script, if so how can I find out?

System info : ubuntu server with 2GB of free ram. And 30 gb free memory. CPU usage was also normal at the time of failure.

With no error in logs I am totally confused.

Suhaib Janjua
  • 3,538
  • 16
  • 59
  • 73
Jya
  • 763
  • 6
  • 22
  • inode count on server? – Jaak Kütt Apr 09 '15 at 11:43
  • Post so code just in case. – Musk Apr 09 '15 at 12:51
  • @musk very sorry I can't do that, legal reason. 2 developers already looked through it. was testing it full day. Didn't find any issue. – Jya Apr 10 '15 at 08:28
  • when you figure it out, send it to [The Daily WTF](http://thedailywtf.com) – tbodt Apr 10 '15 at 19:38
  • @tbodt :-) I probably will – Jya Apr 11 '15 at 04:36
  • It already happened to me that a php function silently fail and abort a script... It had a good reason to fail but not to do it quietly... To catch the problem i found no other solutions than launching the script again and again with the same input parameters and always more and more traces... – mathieu Apr 11 '15 at 05:19
  • @mathieu ya I am trying that, no other way I guess. Which php function failed you ? Maybe it can give me some clues – Jya Apr 11 '15 at 05:37
  • I'm sorry but it's an old problem, i cannot remember precisely. I think it was a complex function (like to encode or to decode), which had vicious inappropriate input parameters. – mathieu Apr 11 '15 at 05:44

0 Answers0