How to prevent Git client hanging during post-receive hook

Question

I have a Git repository that contains a collection of config files and python scripts.

On the server repository, I have a post-receive hook that will:

checkout the master branch,
setup a python virtualenv,
install the pip requirements,
and run some scripts

Like this:

#!/bin/bash
while read oldrev newrev ref; do
    branch=$(git rev-parse --symbolic --abbrev-ref $ref)
    if [[ $branch == "master" ]];
    then
        echo "Ref $ref received in branch $branch"
        echo "setup..."

        cd /srv/repo.git
        git --work-tree=/opt/work-dir checkout master -f

        cd /opt/work-dir
        echo "install virtual environment..." > "setup.log"
        python3 -m virtualenv venv >> "setup.log"
        echo "activate virtual environment..." >> "setup.log"
        source venv/bin/activate >> "setup.log"
        echo "install requirements..." >> "setup.log"
        pip install -r requirements.txt >> "setup.log"
        echo "done." >> "setup.log"

        echo "run scripts..."
        python source/some_script.py --param=value
    else
        echo "Ref $ref received, in branch $branch"
        echo "no action needed."
    fi
done

All of this works fine, but the Git client hangs while the hook is running and will remain like that until I manually cancel. To be clear, the server-side hook does complete successfully, but the client stops logging part way through.

I am aware that I can fix the hanging issue by moving the python setup and execution to a separate bash file and using nohup or disown so that it will continue executing once the terminal closes. However, this would require that I log the result of the script execution separately rather than having it display in the git push log.

What is causing the git client to hang?
How can I monitor the status of the script executions in the git client, without it causing the client to hang?

Calling directly

If I simulate a push by calling the post-receive hook manually, the hook will complete successfully.

hooks/post-receive <<MARK
09998130e13827a097797ff2fd3a973e91665960 fcb0b23a62f47bb73d6ccf2e6bfce324a04eeace master
MARK

Child Processes

I have inspected the child-processes for the hook to see if something isn't exiting properly. As far as I can tell, all child processes exit fully before the parent script continues.

I called ps auxf after every command in the post-receive hook to see what the process tree looked like, and it always returned:

root       711  0.0  0.3  95184  6812 ?        Ss   Feb16   0:00 sshd: username [priv]
username   720  0.0  0.2  95184  4660 ?        S    Feb16   0:04  \_ sshd: username @pts/1,pts/0,pts/2
username   725  0.0  0.1  19892  3712 pts/0    Ss   Feb16   0:00      \_ -bash
root       842  0.0  0.1  49484  3676 pts/0    S    02:49   0:00      |   \_ sudo -i
root       843  0.0  0.1  19940  3768 pts/0    S    02:49   0:00      |       \_ -bash
root      4616  0.0  0.1  50892  3428 pts/0    S    13:59   0:00      |           \_ su - git
git       4617  0.0  0.1  19920  3752 pts/0    S    13:59   0:00      |               \_ -su
git       5108  0.0  0.1  11256  3016 pts/0    S+   17:26   0:00      |                   \_ /bin/bash hooks/post-receive
git       5130  0.0  0.1  38456  3328 pts/0    R+   17:27   0:00      |                       \_ ps auxf

The nest is large because I am logged in over ssh, and sudo'd to root, and then su'd to the git user... but the important part is in the last two lines.

Process of Elimination

I have identified the line that causes the hang:

python3 -m virtualenv venv >> "setup.log"

If I remove this line from the hook, and setup the virtual environment manually before pushing the update, then the push will complete successfully.

For the record, I can confirm that running the python setup in a background script stops the client hanging. I have an ok(ish) workaround but I'd appreciate it if someone could tell me what is causing the client to hang. — , Feb 17 '19 at 13:10

score 1 · Answer 1 · answered Feb 17 '19 at 17:00

1

I believe that either python source/some_script.py --param=value or pip install leaves a hanging child process.

From my experiments, I guess git has some server-side logic to wait for all the children processes left over by the post-receive script. It waits for a hanging child forever.

Please verify your prediction that the post-receive script always completely finishes: run manually post-receive in the shell, then look up the children with ps

answered Feb 17 '19 at 17:00

kubanczyk

13,812
5
41
55

thanks for the pointers, I've updated my question with the latest. Seems to be something to do with `python3 -m virtualenv venv `; works fine when run from bash locally, but causes the git client to hang during a `push`. – Feb 17 '19 at 17:47
I think `ps aufx` does not trace properly orphaned grandchilds whose parent have already finished. Look for "strange" processes that also have the same terminal (`pts/0` for example). @Marvin – kubanczyk Feb 17 '19 at 18:32
It seems that the problem is a timeout issue for the client... if the server-side hook is quiet for more than a few seconds, then the git client concludes "The remote end hung up unexpectedly". I have set more verbose logging messages and it now runs successfully seemingly because it has a more constant stream of messages. – Feb 17 '19 at 22:09

How to prevent Git client hanging during post-receive hook

Calling directly

Child Processes

Process of Elimination

1 Answers1

Linked