2

We had an OutOfMemoryException in our logstash 5.6.6 which lead to the checkpoint. head file not to be written properly:

ls -l /var/lib/logstash/queue/main/
total 266424
-rw-r--r-- 1 logstash logstash        34 Mar  1 20:33 checkpoint.55779
-rw-r--r-- 1 logstash logstash         0 Mar  1 20:34 checkpoint.head
-rw-r--r-- 1 logstash logstash 262144000 Mar  1 20:33 page.55779
-rw-r--r-- 1 logstash logstash 262144000 Mar  1 20:34 page.55780

Is there a way to repair the checkpoint.head or at least to get page 55779 to be loaded into the pipeline so we don't loose a couple thousand events?

Dero
  • 185
  • 1
  • 14

1 Answers1

1

We ended up having a look at a working checkpoint.head file and modify it to our needs. In hex a file may look like this:

00000000: 0001 0000 d9e4 0000 d9e3 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 1855  ...............U
00000020: 2b6b 0a                                  +k.

What needs to be edited are:

  • Byte 3-6: 32 bit signed integer: number of newest page file
  • Byte 7-10: 32 bit signed integer: number of lowest not-processed page file
  • Byte 31-34: 32 bit signed integer: hash value

All numbers are big endian.

To get the hash just set the values for page files and start up logstash. It will show you an error message mentioning the expected and actual hash, replace the value in checkpoint.head with the actual has from that error message.

Dero
  • 185
  • 1
  • 14