Unusual server crash on EC2, syslog has line of ^@^@^@^@^@

Question

Trying to get at the root of why a server went down for 20 minutes, I looked at the syslog for that time period and saw this:

Jan  3 07:50:01 tools CRON[17085]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jan  3 07:55:01 tools CRON[17773]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jan  3 07:55:01 tools CRON[17774]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Jan  3 08:19:44 tools kernel: imklog 4.2.0, log source = /proc/kmsg started.
Jan  3 08:19:44 tools rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="470" x-info="http://www.rsyslog.com"] (re)start

EC2 shows this CPU utilization during the time of the crash (in the middle, before 13:00 UTC).

EC2 monitoring details CPU usage

So there is nothing even visible during that time period. None of our other EC2 instances crashed and I can find no evidence that it was an application error. In fact, this happened on our tools server (apache, mongodb, and redis). Monit was also running, but there are no suspicious logs at the time of the crash.

What could have caused this crash and what do the ^@'s mean in the syslog?

This `^@` is just a funky representation of binary `0`. It is used in some viewers/editors. No idea why string of zeroes appeared in your syslog, not an usual thing. — kubanczyk, Jan 03 '12 at 17:28

score 1 · Accepted Answer · edited Jun 11 '20 at 10:02

The answer came on the AWS forum: https://forums.aws.amazon.com/message.jspa?messageID=308434

Reed,

There was a problem with the underlying hardware that caused the underlying system to crash. The artifact of the syslog lines, could be just a representation that the system was in mid write when it crashed, or the lines could have come from the the cron process that was running on the previous line.

Nathan

Unusual server crash on EC2, syslog has line of ^@^@^@^@^@

1 Answers1