2

I rendered unusable one of my (linux 2.6) remote systems by running on it a process that put the system in an awful state (is either hogging the CPU, consuming too much memory, or exhausting some other system resource). I can no longer login via ssh (I get "Write failed: Broken pipe") or telnet ("connection refused").

I am able to ping the system. Also I am, curiously, able to login via FTP, including the ability to get, put and delete files. Although I have superuser's password, I cannot login as root via FTP because the system has that capability disabled. I can only login under my normal user account.

I know the pid of the offending process.

Here is my question: Given the above information, is there a way to kill the rogue process via an FTP login under the same user from which the process is running? I don't need it to be a graceful kill. For example, by deleting something in /var/proc/ that would cause the process to crash (though I tried that and the delete failed)

-bb

ePhrygian
  • 51
  • 6
  • Is, by any chance, this process running in a cgroup? And would you have write-access to this cgroup? You could try to restrict the CPU usage to a critically low level. – Martin Richard Feb 25 '14 at 17:07
  • is it possible to upload some php script ? run it though the browser ? – exussum Feb 25 '14 at 17:21
  • #-- I checked /cgroup and everything in that tree is owned by root and not world-writable :( – ePhrygian Feb 25 '14 at 18:11
  • #-- for running php script, UserDir is disabled for apache and the html doc directories are not writable by my uid :( – ePhrygian Feb 25 '14 at 18:12
  • Any chance you can connect with KVM? What about remote reboot? If not, mail your host and ask them to do a hard restart of the box. – Mike Feb 27 '14 at 07:14

2 Answers2

1

I couldn't post a comment because I am new here but since you didn't put this info within your question, here are some things you should try if at all possible:

  • Is the system a virtual remote server running on a host such as VMware or Xen? If so, can you get access to the host client to manage the VM?
  • Is the system a physical server? Is so, does it have DRAC or iLO configured or can you get someone to have physical access to set this up? This would then mean you'd have to boot into the BIOS of either of the two features, which would solve your initial problem. So this brings another question...
  • If the server is a physical server, do you have access to another server or computer close to this one where you can hookup a serial cable to it and use Minicom to access the server via the console as if you were physically there?

Sounds like you have FTP secured in a chroot environment, which is a good thing. So, you will most likely not be able to get passed your jailed account.

From my experience, when there is a high CPU and my SSH connection renders inoperable, it does have a chance of recovering or giving you enough resource to allow new connections to be made. With this in mind, you could probably write a small script (SHELL Script) that attempts to login for you to that server every minute or every few minutes until successful and if successful, have it kill the process since you know the PID. The SSH command (client) does allow you to pass commands. If the connection is successfully and does send the kill command successful, that your client machine capture the exit status and send you an alert to your phone letting you know it did what you asked.

Tito Valentin
  • 147
  • 1
  • 9
1

If you have write permissions to /etc/cron.d/ you could put in a cronjob that can run whatever you want as any user you want.

Red Cricket
  • 9,762
  • 21
  • 81
  • 166