test: test-cluster-send-handle-large-payload intermittent failures

[matthewloring]

• Version: master
• Platform: OS X
• Subsystem: Test

This test seems to be timing out sometimes. Introduced by #14588.

\cc @addaleax

[addaleax]

Hm, I ran a stress test on the original PR because it had been flaky before, but that came back green, and a local stress test worked fine for me too, so I assume this is something platform-specific. So, uh, here are two stress tests to narrow that down a bit:

OS X: https://ci.nodejs.org/job/node-stress-single-test/1367/
Linux: https://ci.nodejs.org/job/node-stress-single-test/1368/

[matthewloring]

I think @bnoordhuis has experienced this too (#14730 (review)). He may have suggestions on reproducing it. For me it happens fairly infrequently (~1 in 20 runs).

[addaleax]

@nodejs/platform-macos … I don’t have access to OS X myself, but I am available to anybody who wants me do help debug my mistakes ;)

[Trott]

I can replicate this locally. This might work on whatever your local OS is as well to replicate it:

$ tools/test.py -j 96 --repeat 192 test/parallel/test-cluster-send-handle-large-payload.js 
=== release test-cluster-send-handle-large-payload ===                    
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
=== release test-cluster-send-handle-large-payload ===                   
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
=== release test-cluster-send-handle-large-payload ===                   
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
=== release test-cluster-send-handle-large-payload ===                   
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
=== release test-cluster-send-handle-large-payload ===                   
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
=== release test-cluster-send-handle-large-payload ===                   
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
=== release test-cluster-send-handle-large-payload ===                   
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
=== release test-cluster-send-handle-large-payload ===                   
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
=== release test-cluster-send-handle-large-payload ===                   
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
=== release test-cluster-send-handle-large-payload ===                   
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
=== release test-cluster-send-handle-large-payload ===                   
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
=== release test-cluster-send-handle-large-payload ===                   
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
=== release test-cluster-send-handle-large-payload ===                   
Path: parallel/test-cluster-send-handle-large-payload
Command: out/Release/node /Users/trott/io.js/test/parallel/test-cluster-send-handle-large-payload.js
--- TIMEOUT ---
[02:11|% 100|+ 179|-  13]: Done 
$

First 96 tests pass in about 11 seconds, and then it pauses for two minutes as some of the tests (13 above) time out.

Easiest/fastest solution would be to move the test to sequential, but of course that might not be the best solution.
¯\(ツ)/¯

[addaleax]

Easiest/fastest solution would be to move the test to sequential, but of course that might not be the best solution.

Every time somebody proposes that, I think either the test is broken or Node is broken. ;) I don’t think there’s a reason to assume this test should be flaky under load.

[addaleax]

Also, no, no luck reproducing under Linux, no matter how many parallel jobs.

[Trott]

Also, no, no luck reproducing under Linux, no matter how many parallel jobs.

Looks like -j doesn't even have that much to do with it other than making it happen more reliably. I'm seeing it with just tools/test.py -j 1 --repeat 192 test/parallel/test-cluster-send-handle-large-payload.js

[addaleax]

@Trott Any chance you could do some digging and figure out whether it’s the parent or the child process that’s hanging? And which handles are keeping it open (assuming it’s stuck with 0 % CPU, not 100 % CPU)?

[Trott]

@addaleax Yes, looking right now...might have to stop for a few meetings, but will pick up again later if so...

[Trott]

When this times out, the process.send() in the subprocess is getting called, but the message event on worker is not being emitted (or at least the listener is not being invoked).

If I add a callback to process.send(), it never indicates an error in this test, whether the test succeeds or times out.

Judging from #6767, process.send() may be fire-and-forget. Maybe the message gets received and maybe not.

I don't know why macOS would be more susceptible to missing the message than anything else. If this is not-a-bug behavior, I guess we can add a retry. If this is a bug... ¯\(ツ)/¯

The .send() from the parent process to the subprocess never seems to fail.

[addaleax]

Okay, so this might be fix-able by adding an extra round-trip. I can try that on the weekend if nobody beats me to it.