After a short period of usage thinking sphinx on the staging server I have found strange behavior. Every 2-3 days search on the website becomes unavailable because search daemon is down. When I try to start it manually with rake ts:start command it fails and suggest me to see searchd.log file. I have found these at the end of the log file:
[Thu Aug 21 11:17:27.754 2014] [11359] binlog: replaying log /var/www/ultimate-staging/shared/log/binlog/binlog.001
[Thu Aug 21 11:17:27.754 2014] [11359] FATAL: binlog: update: descending tid (index=(null), lasttid=2, logtid=1, pos=135)
[Thu Aug 21 11:17:27.756 2014] [11358] Child process 11359 has been finished, exit code 1. Watchdog finishes also. Good bye!
After Googling some time I found similar issue described which were solved by cleaning binlog folder. Its helped me too, but after 3 days later search daemon raised the same error and I've deleted files from binlog folder again. Is anybody knows how to fix this issue forever? What's the cause of this weird behavior? Maybe I missed something when tuned shpinx?
There are some other info within searchd.log file that I've found before searchd was shut down:
[Mon Aug 18 17:25:11.616 2014] [21780] rotating indices (seamless=1)
[Mon Aug 18 17:25:11.641 2014] [21780] rotating index 'tour_delta': started
[Mon Aug 18 17:25:11.641 2014] [21780] rotating index 'tour_delta': success
[Mon Aug 18 17:25:11.641 2014] [21780] rotating index 'tour_core': started
[Mon Aug 18 17:25:11.643 2014] [21780] rotating index 'tour_core': success
[Mon Aug 18 17:25:11.643 2014] [21780] rotating index 'point_delta': started
[Mon Aug 18 17:25:11.643 2014] [21780] rotating index 'point_delta': success
[Mon Aug 18 17:25:11.643 2014] [21780] rotating index 'point_core': started
[Mon Aug 18 17:25:11.654 2014] [21780] rotating index 'point_core': success
[Mon Aug 18 17:25:11.655 2014] [21780] rotating index: all indexes done
[Mon Aug 18 17:30:17.536 2014] [21780] rotating indices (seamless=1)
[Mon Aug 18 17:30:17.583 2014] [21780] rotating index 'tour_delta': started
[Mon Aug 18 17:30:17.589 2014] [21780] rotating index 'tour_delta': success
[Mon Aug 18 17:30:17.589 2014] [21780] rotating index 'tour_core': started
[Mon Aug 18 17:30:17.592 2014] [21780] rotating index 'tour_core': success
[Mon Aug 18 17:30:17.593 2014] [21780] rotating index 'point_delta': started
[Mon Aug 18 17:30:17.598 2014] [21780] rotating index 'point_delta': success
[Mon Aug 18 17:30:17.599 2014] [21780] rotating index 'point_core': started
[Mon Aug 18 17:30:17.611 2014] [21780] rotating index 'point_core': success
[Mon Aug 18 17:30:17.611 2014] [21780] rotating index: all indexes done
[Mon Aug 18 17:30:36.078 2014] [21780] caught SIGTERM, shutting down
[Mon Aug 18 17:30:36.229 2014] [21780] shutdown complete
[Mon Aug 18 17:30:36.246 2014] [21779] Child process 21780 has been finished, exit code 0. Watchdog finishes also. Good bye!
[Mon Aug 18 17:30:36.491 2014] [ 4410] Child process 4411 has been forked
[Mon Aug 18 17:30:36.493 2014] [ 4411] listening on 127.0.0.1:9306
[Mon Aug 18 17:30:36.497 2014] [ 4411] binlog: replaying log /var/www/ultimate-staging/shared/log/binlog/binlog.001
[Mon Aug 18 17:30:36.497 2014] [ 4411] FATAL: binlog: update: descending tid (index=(null), lasttid=2, logtid=1, pos=135)
[Mon Aug 18 17:30:36.499 2014] [ 4410] Child process 4411 has been finished, exit code 1. Watchdog finishes also. Good bye!
I have tuned reindexation with cron jobs every 5 minutes. Maybe it's failing because reindexation is running too often. Is it possible?
