7

I'm hosting an experimental/testing Linux box, running Debian Wheezy 7.4.0 distribution. Different users log into the machine over ssh to their accounts and are allowed to run the development tools and leave their programs running as services in background if they so wish.

Since this is a testing machine for all kinds of purposes there is often a need to restart the whole machine and then the users have to log back in and restart their user-space stuff that was running. I would like to automate that. Basically I would like to provide the users with a mean to launch stuff right after the machine boots up(after everything else is initialized) and a mean to launch stuff upon system shutdown(with no time limitations, basically stalling the shutdown until all those shutdown user processes have completed).

What I have tried so far:
I've created an init bash script, by following the principles found in the 'skeleton' template file under /etc/init.d/ (Skeleton template source code: https://gist.github.com/ivankovacevic/9917139)

My code is here: https://github.com/ivankovacevic/userspaceServices

Basically the script goes through users home directories and looks for executable files in corresponding subdirectories named .startUp, .shutDown or .status. Depending on the event that is currently going on the scripts get executed with su as if the users have started them themselves.

The problem I'm currently facing with this approach is that there is a strange process left hanging after the system boots and the script starts all the processes of other users. This is how it looks in the processes list:

UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root      3053     1  0  1024   620   1 17:42 ?        00:00:00 startpar -f -- userspaceServices

I don't know what that process is and man page for it does not mention the -f argument. So I'm clueless but I must be doing something wrong since no other script/service from init.d leaves such a process hanging after boot.

So I'm looking for someone to help me debug this solution I have(which also seems a bit complex in my opinion). Or give me some idea how this could be implemented in an entirely different way.

UPDATE
I've started a separate question for the startpar issue: startpar process left hanging when starting processes from rc.local or init.d

UPDATE 2
Problem solved for my original solution. Check the previously mentioned question for startpar. The code on GitHub is also corrected to reflect that.

UPDATE 3 - How to use crontab
As Jenny suggested, regular users can schedule tasks to be executed once upon boot using crontab. I find that to be the easiest method, if all you need is starting user tasks on boot and not shutdown. However there is a drawback that users can leave cron process "hanging" as parent when they launch on-going, service-like tasks. First let me just explain how it works:

regular users themselves should call:

crontab -e

( -e as in edit ) Which opens a default console text editor with their user crontab file. To add a task to be executed at boot, a user must add one line at the end of the file:

@reboot /path/to/the/executable/file

Now if the user would do just that and if that file is not just some simple script that linearly completes something and ends, but some sort of watchdog for example, after a reboot you would end with something like this in your processes list:

    1  2661 root       20   0 20380   860   660 S  0.0  0.0  0:00.00 ├─ /usr/sbin/cron
 2661  2701 root       20   0 33072  1152   868 S  0.0  0.0  0:00.00 │  └─ /USR/SBIN/CRON
 2701  2944 someuser   20   0  4180   580   484 S  0.0  0.0  0:00.00 │     └─ /bin/sh -c ./watchdog
 2944  2945 someuser   20   0 10752  1204  1016 S  0.0  0.0  0:00.00 │        └─ /bin/bash ./watchdog
 2945  2946 someuser   20   0 23696  4460  2064 S  0.0  0.1  0:00.01 │           └─ /usr/bin/python ./some_program.py

To avoid that the user needs to modify his crontab entry to look like this:

@reboot /path/to/the/executable/file >/dev/null 2>&1 &

The redirections of file descriptors are optional but recommended to keep it clean. If you want to study why, try looking at them:

ls -l /proc/pid_of_started_process/fd
Ivan Kovacevic
  • 1,801
  • 3
  • 15
  • 19

1 Answers1

8

I agree that your solution seems a bit complex, so I'll go with "give me some idea how this could be implemented in an entirely different way" :-)

  1. The standard solution for this is to use a configuration management system, such as puppet, and allow users to add their stuff to the puppet config for the server. Puppet will then push out the start script and add them to the relevant runlevels.

  2. A quicker way would be to give them sudoedit access to /etc/rc.d/rc.local and add their things there.

  3. Or give them each a directory to put the start scripts they want started, and have a cron job copy those scripts to /etc/init.d, inserting su $USER -c at suitable places and run chkconfig on them.

  4. Or give them each a directory to put the start scripts, and add some lines at the end fo /etc/rc.d/rc.local to go through those directories and run edited su $USER -c 'script start' on each script in them.

Edited to add: 5. Let them use crontab to schedule the jobs to be run @reboot

Jenny D
  • 27,780
  • 21
  • 75
  • 114
  • The issue with your suggestions is that they allow users to edit scripts which would be executed as root during boot or shutdown. Thus without me inspecting these scripts every time the machine needs to reboot, I can not be sure that some code provided by users will not modify(harm) the system in unwanted ways. – Ivan Kovacevic Apr 01 '14 at 17:51
  • That constraint wasn't in the spec... You could still do that with puppet, either by checking scripts before deploying them or by changing puppet config to deploy those scripts to run as user. But I also changed point 3 and 4 and added a point 5 in response to that. – Jenny D Apr 01 '14 at 18:05
  • :) thanks for your edits, don't take it too personal :). Puppet seems just another layer of complexion to the topic. The number 4 idea stands. That's what I'm testing right now. BUT I still see this problem with startpar -f process when I do that. Now I see it in the processes list as: startpar -f -- rc.local and all I have added in rc.local is one line to start a simple watchdog bash script in one of the users directory exactly as per your number 4. suggestion. I now really would like to figure out what the heck is that process. Maybe I should start another separate question for that... – Ivan Kovacevic Apr 01 '14 at 18:10
  • I forgot to say, crontab @reboot seems like a nice solution. However I would also like something right before shutdown. Can crontab be used for that also? – Ivan Kovacevic Apr 01 '14 at 18:15
  • I don't take it personal, I enjoy figuring things out which is why I hang around here... Yes, puppet adds complexity, so when you're using just one server it may be too much - on the other hand, this kind of thing is one of the things it's meant for. – Jenny D Apr 01 '14 at 18:18
  • 1
    as for crontab at shutdown - no, it doesn't have that. But you could add a K99 script that looks at every user's crontab, checks for @reboot and runs the same script with `su -c $USER 'crontabentry stop'` - a little contrived, but doable. – Jenny D Apr 01 '14 at 18:19
  • I should add that this is a fun problem, probably more so for me who doesn't get her server cracked if it comes out wrong than for you :-) – Jenny D Apr 01 '14 at 18:20
  • hahaha, yup. And this with K99 script for shutdown could work, thanks for the ideas! Now I'm basically just wondering what the heck is(was) that startpar -f and why I can not avoid it like any other normal init script does. I take that personal! :D – Ivan Kovacevic Apr 01 '14 at 18:24
  • Aha yes, I also wanted to ask you, will crontab @reboot job work also in case of shutdown or system crash? I found some comment on one site that it works only after an issued reboot?! Seems nonsensical to me. – Ivan Kovacevic Apr 01 '14 at 18:25
  • It should work at every boot, not just an issued reboot. I have never heard of a distribution where it didn't work that way. – Jenny D Apr 01 '14 at 18:33
  • Thanks for all the effort you've put into this answer. I have accepted it. I've opened a new question regarding startpar: http://serverfault.com/questions/585975/startpar-process-left-hanging-when-starting-processes-from-rc-local-or-init-d – Ivan Kovacevic Apr 01 '14 at 18:52
  • 1
    Thanks for the link, it was interesting to know the cause! (I've upvoted your answer - it's not unusual that non-redirected output causes a process to not be successfully daemonized, but I didn't know enough about startpar to pick that up immediately. The next person who finds that question will definitely be helped by your answer.)¨ – Jenny D Apr 02 '14 at 13:59
  • Thanks for the up-vote! :) I've also added a little update to my question here, regarding the crontab method. – Ivan Kovacevic Apr 02 '14 at 22:47