Datastore emulator orphans child process when run in background

[dhermes]

I noticed this when running tox -e datastore-emulator in gcloud-python.

$ date "+%H:%M:%S"
11:47:18
$ ps
  PID TTY          TIME CMD
25498 pts/9    00:00:05 bash
26411 pts/9    00:00:00 ps
$ gcloud beta emulators datastore start &> log.txt &
[1] 26433
$ date "+%H:%M:%S"
11:47:29
$ ps
  PID TTY          TIME CMD
25498 pts/9    00:00:05 bash
26433 pts/9    00:00:00 gcloud
26442 pts/9    00:00:00 python2
26472 pts/9    00:00:00 bash
26474 pts/9    00:00:00 java
26486 pts/9    00:00:02 java
26515 pts/9    00:00:00 ps
$ date "+%H:%M:%S"
11:47:39
$ kill 26433
[1]+  Terminated              gcloud beta emulators datastore start &> log.txt
$ date "+%H:%M:%S"
11:47:49
$ ps
  PID TTY          TIME CMD
25498 pts/9    00:00:05 bash
26442 pts/9    00:00:00 python2
26472 pts/9    00:00:00 bash
26474 pts/9    00:00:00 java
26486 pts/9    00:00:02 java
26551 pts/9    00:00:00 ps

/cc @jgeewax @tseaver

[eddavisson]

@vilasj I think this issue applies to any emulator in the gcloud emulator group that is started as a subprocess.

[dhermes]

I can confirm it also holds creates orphans for pubsub. I'm addressing for now by manually corralling all the child processes via psutil

proc = psutil.Process(pid)
for child_proc in proc.children(recursive=True):
    try:
        child_proc.kill()
        child_proc.terminate()
    except psutil.NoSuchProcess:
        pass
proc.terminate()
proc.kill()

[vilasj]

Note that the top level parent process here is gcloud.sh, which starts a python process that executes gcloud.py, which starts a bash process that executes gcd.sh, which starts a java process which is the actual emulator!

When you send SIGTERM to the background parent process, it just kills itself. It does not propagate the SIGTERM to its children. That is why you still see the child processes running.

If you want to kill the whole process tree, you should use kill -- -[PID] as described here:

http://stackoverflow.com/questions/392022/best-way-to-kill-all-child-processes
http://stackoverflow.com/questions/976059/shell-script-to-spawn-processes-terminate-children-on-sigterm

Or alternatively, you can kill the child emulator process.

So in your example the following would work:

kill -- -26433

or

kill 26474

[dhermes]

That's great. I had a PR out to add Python code that manually kills the child processes and have updated it with the recommendation to use kill -- -${PID}.

This helps the users of gcloud-python that manage to read our CONTRIBUTING, but doesn't do much for other users of the emulators.

Do you expect a typical user to know they need to clean up all children after

gcloud beta emulators foo start &

[vilasj]

This is not specific to gcloud beta emulators. It happens when you background any long running process that starts sub-processes.

Perhaps we can put something in the help text for the command about running the emulators in the background. I have created an internal bug for this.

[dhermes]

Yes I realize (did I give an indication that I thought child processes were specific to gcloud beta emulators?).

But I'm not concerned about me, I'm concerned about other users.

Why would it be clear to the average user that the process spawns 5 more processes? The fact that start has no option to be background-ed makes users do it manually and in turn makes them orphan the subprocesses they didn't know about.

[vilasj]

Of course, this is not about you or me, it is about our users. Did you read the second sentence of my comment?

I agree that it may not be clear to users that start creates sub-processes. That is why I have filed a bug to add help text to clarify this.

The alternative is to run all emulators within the same process. This is not a reasonable alternative in this case because gcloud is written in Python and the emulators are written in Java.

[dhermes]

Great, that's a start. You should trap the SIGINT/SIGTERM/SIGKILL signal(s) for the parent and try to clean up the children (I used psutil in Python to do this in a cross-platform way, though could've used Python's atexit as well).

It's unclear why being able to background the emulator and send the output to a log file is not a feature.

Closing this issue out since we are no longer chatting about google-cloud-datastore.

[vilasj]

Agree with all your suggestions.

• We can look into trapping signals in gcloud.sh and performing some sort of cleanup.
• We hope to add backgrounding capability in the future along with emulator 'process management'.

[dhermes]

Good to hear!