4

OK so I have paramiko v2.2.1 and I am trying to login to a machine and restart a service. Inside the service scripts it basically starts a process via nohup. However if I allow paramiko to disconnect as soon as it is done the process started terminates with a PIPE signal when it writes to stdout.

If I start the service by ssh'ing into the box and manually starting it there is no issue and it runs in the background fine. Also if I add long sleep 10 before disconnecting (close) paramiko it also seems to work just fine.

The service is started via a init.d script via a line like this:

env LD_LIBRARY_PATH=$bin_path nohup $bin_path/ServerLoop.sh \
  "$bin_path/Service service args" "$@" &

Where ServerLoop.sh simply calls the service forever in a loop like this so it will never die:

SERVER=$1
shift
ARGS=$@
logger $ARGS
while [ 1 ]; do
    $SERVER $ARGS
    logger "$SERVER terminated with exit code: $STATUS. Server has been restarted"
    sleep 1
done

I have noticed when I start the service by ssh'ing into the box I get a nohup.out file written to the root. However when I run through paramiko I get no nohup.out written anywhere on the system ... ie this after I manually ssh into the box and start the service:

root@ts4700:/mnt/mc.fw/bin# find / -name "nohup*"
/usr/bin/nohup
/usr/share/man/man1/nohup.1.gz
/nohup.out

And this is after I run through paramiko:

root@ts4700:/mnt/mc.fw/bin# find / -name "nohup*"
/usr/bin/nohup
/usr/share/man/man1/nohup.1.gz

As I understand it nohup will only redirect the output to nohup.out if "If standard output is a terminal" (from the manual), otherwise it thinks it is saving the output to a file so it does not redirect. Hence I tried the following:

In [43]: import paramiko

In [44]: paramiko.__version__
Out[44]: '2.2.1'

In [45]: ssh = paramiko.SSHClient()

In [46]: ssh.set_missing_host_key_policy(AutoAddPolicy())

In [47]: ssh.connect(ip, username='root', password=not_for_so_sorry, look_for_keys=False, allow_agent=False)

In [48]: stdin, stdout, stderr = ssh.exec_command("tty")

In [49]: stdout.read()
Out[49]: 'not a tty\n'

So I am thinking that nohup is not redirecting to nohup.out when I run it through paramiko because tty is not returning a terminal. I don't know why adding a sleep(10) would fix this though as the service if run on the command line is quite verbose.

I have also noticed that if the service is started from a manual ssh its tty in the ps ax output is still set to the ssh tty ... however if the process is started by paramiko its tty in the ps ax output is set to "?" .. since both processes are run through nohup I would have expected this to be the same.

If the problem is that nohup is indeed not redirecting the output to nohup.out because of the tty is there a way to force this to happen or a better way to run this sort of command via paramiko?

Thanks all, any help with this would be great :)

pynexj
  • 19,215
  • 5
  • 38
  • 56
othane
  • 558
  • 5
  • 15
  • 1
    try `ssh.exec_command(command_to_start_the_service, get_pty=True)` when starting the service. – pynexj Aug 09 '17 at 03:06
  • @whjm that seem to have done the trick, thanks!! .. for anyone interested, I have found redirecting stdout and stderr to /dev/null in nohup seem to work too which is a little better as it is not client specific (we have several code bases calling these services, some in .net etc so they may have similar issues) – othane Aug 09 '17 at 04:26
  • there is still one thing I don't get ... why does this work ok with the sleep ... I mean with or without it paramiko is not providing proper pty's hence nohup fails to redirect stdout etc .. so I don't get how this process doesn't always raise a SIGPIPE since it constantly spews output to the terminal all day long? ... BTW its this code has been running for years and this problem has only just shown up with a slight timing change – othane Aug 09 '17 at 04:38
  • could you paste the exact python func call (or the full paramiko script) you used to restart the service? and where did you put the `sleep(10)` as you tried? when will you call `ssh.close()`? – pynexj Aug 09 '17 at 06:48
  • unfortunately the code is quite large and spread across many files .. also some of it is confidential so I cannot easily post that .. however I will attempt to make something more succinct that will show the issue. – othane Aug 09 '17 at 10:09

0 Answers0