1

Possible Duplicate:
My server's been hacked EMERGENCY

We've got a production server running Ubuntu 11.10. We're encountering segfaults that appear under various conditions. The simplest reproducible case is when we login to an ssh session as our administrative user and enter a bogus command. You'd expect the standard "command not found" error message. Instead, we get a segfault in python. The user's default shell is /bin/bash.

For example:

$ asdf
Segmentation fault

Info from /var/log/syslog:
Jul  6 15:39:20 PROD001 kernel: [2155960.605695] python[7873]: segfault at 0 ip (null) sp 00007fffd030b808 error 14 in python2.7[400000+233000]

Some details about the server:

$ uname -a
Linux PROD001 3.0.0-16-server #29-Ubuntu SMP Tue Feb 14 13:08:12 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/issue
Ubuntu 11.10 \n \l

Before we ask the IT department to reinstall the O.S., I'd like to understand what got us here. The system and/or this particular user's environment is suspect. Many people have touched this server over the past year, so I'm wondering if it is missing libraries, incorrectly installed packages, etc. I'm hoping that if we can understand what's going wrong in this case, it will help explain why we're getting segfaults in a couple of other scenarios.

Any tips on troubleshooting this segfault will be appreciated!

Community
  • 1
Alan
  • 541
  • 1
  • 6
  • 20

2 Answers2

3

Ubuntu implements a command-not-found function in /etc/bash.bashrc that calls a python script to provide more helpful suggestions than the default "command not found" error message.

Perhaps the package or some dependencies need to be re-installed. A simple test to see if this package is at fault would be to remove the command-not-found package and see if you still get the segfault:

apt-get remove command-not-found

Jerf Altair
  • 54
  • 2
  • 1
    Unix newbie here, so thanks for the clarification! Given the questionable state of this production box, we've decide to re-install things anyway. – Alan Jul 09 '12 at 17:26
  • That's probably much easier than figuring out and repairing the root cause. – Jerf Altair Jul 09 '12 at 17:57
2

You type a command on the shell, and get a segfault... You're expecting the shell to be bash, but it gets executed as python. That's scary shit, right there.

Disconnect that machine from the network, and have it reinstalled ASAP. Restore any local files from a known-good backup, because it looks a bit like that machine might've been rootkitted.

I'm probably wrong, but erring heavily on the side of caution, because it looks like /bin/bash isn't what you expect it to be.

Tom O'Connor
  • 27,480
  • 10
  • 73
  • 148
  • 1
    How is that "scary shit"? That's just Ubuntu's command-not-found doing it's thang, and failing miserably. There's something wrong, sure, but I wouldn't leap immediately to "OMG I'm h4xored!". – womble Jul 07 '12 at 03:10