Add output option to JSON reporter (#4131)

[dorny]

Extends JSON reporter with option to write output directly to file instead of a console.

closes #4131

Description of the Change

• If output option was provided to JSON reporter:
  • On EVENT_RUN_END it will write JSON output to given file - single call to fs.writeFileSync()
  • Directory structure will be recursively created if needed - fs.mkdirSync()
  • If test runs in browser it will throw unsupported error
• Unit tests has been updated
  • Verifies if reporter writes to given file same JSON content as it sets on the runner instance
  • Verifies if reporter throws error if output option is provided and test runs in browser
  • Diff is bigger unless you hide whitespace only changes because existing tests were put into describe() group
• Documentation has been updated

Alternate Designs

xunit reporter uses write stream and produces output continuously during test execution.
Similar behavior is provided by json-stream reporter.
json reporter outputs a single json when the tests have completed so I've used single call to fs.writeFileSync() instead of creating write stream.

I've also considered refactoring of json.spec.js.
Ideally unit test should execute JSONReporter directly instead of using real Runner instance.
Also tests should capture standard output and run assertion on it instead of using testResults property on runner which doesn't look to be used anywhere else.

I've tried to rewrite tests using createMockRunner() from test helpers but it didn't work.
Mock runner doesn't supports simulation of multiple events and reporter state is not preserved between multiple runs.
Therefore it was not possible to use it for json reporter without bigger changes.
I still think refactoring the tests would improve code quality but it is out of scope of this PR.

Why should this be in core?

• It's a common feature, widely supported in other test frameworks (JavaScript or other languages)
• Same option is already supported in xunit reporter
• JSON output is usually processed by another tool and any noise on standard output would break it

Benefits

Compared to redirecting report output to file using shell pipe:

• valid JSON will be written to file even if some test produces noise on standard output
• together with mocha-multi-reporters enables having both console output and file output

[coveralls]

Coverage increased (+0.02%) to 94.403% when pulling eff4daa on dorny:issue/4131-add-json-output-option into bbf0c11 on mochajs:master.

[juergba]

@dorny thank you for this PR.
Yes, json.spec.js could benefit from some refactoring ...
I just have one remark, otherwise looks like a very diligent work.

Edit: the error is swallowed only for a failing test. For a passing test the error is printed to the console. This is a bug in our uncaught exception handler, which I will investigate later.
Edit II: no, it doesn't matter, whether the test fails or passes, the error gets swallowed.

[juergba]

The root of the error being swallowed is, that we have two EVENT_RUN_END listeners, with incorrect order:
a) one is registered within the runner and sets runner.state to stopped (should run first)
b) one is registered within the reporter (runs first, should be second)

So I see three options:

• we use try/catch in json reporter
• we use emitter.prependListener for registering a), but I'm unsure about browser compability.
• we delay the execution of b) with setImmediate()

Edit: or we move the file operations to reporter.done.

[juergba]

Yes, and there is Mocha in parallel mode, quite a recent feature. I'm unsure about the implications, but our docs states:
In parallel mode, reporter output is buffered; reporting will occur after each file is completed.

We have to make sure there will be only one file created.

[dorny]

@juergba thanks for your support. Right now I'm recovering from eye surgery so I've to reduce my screen time significantly for next few days. I will get back to this afterwards. Meanwhile feel free to modify PR yourself if you find solution.

[github-actions]

This PR hasn't had any recent activity, and I'm labeling it stale. Remove the label or comment or this PR will be closed in 14 days. Thanks for contributing to Mocha!

[juergba]

@dorny thank you