Types of Performance Tests

[wesleytodd]

I wanted to start an issue to document the type of performance tests we want to be able to run. Specifically we should be able to relate the "type of test" to a high level metric or goal we want to track.

1. Micro-benchmarks: aim to test individual method implementations. The top level goal of these tests is to enable fast iteration when improving implementations within a single method or small workflow. These should track execution time average over many executions and can likely use a tool like https://jsperf.app/. They are most useful in the context of a PR claiming performance impact, but are not very useful to track longer term. Micro-benchmarks are also well known for showing misleading results, so should be designed with caution.
2. Benchmarks: aim to track key metrics in a system to compare over time. The top level of these goals is to track performance targets and are often used to compare to other frameworks or languages. These can track a broad range of metrics (cpu, memory, execution time, etc) and should be run many times in an isolated environment for best results. They can be used to test small systems all the way up to full frameworks. We can use a tool like https://github.com/RafaelGSS/bench-node for these. They are valuable to track over time, and can help spot trends and regressions.
3. Load Testing: aims to test the full E2E performance of a system, for express this will mean a full server/client interaction over http. The top level goal of these tests is to get a holistic picture of performance impact for different types of workloads which represent real world usage. These should track all available metrics, including system ones out side of the node.js process as well as client side latency and response status. These are valuable to put other types of performance tests into perspective for how they impact the real world and are valuable to track over time.

There are more types, and each of these has some details that are important for us, but I am not sure how much detail we would want our docs/recommendations across the project to go. So for now I will leave it at that to start the discussion. Also, I am sure there are great resources out there we should be linking to instead of re-writing that content here.

In addition to the types of test, I think it is important to call out that we also need diagnostics tooling appropriate for each of these to use while debugging why one test result is faster or slower than the other. And each type will have different needs there. Here are some ideas for this:

1. Micro-benchmarks: CPU profiler, memory profiler, chrome dev tools
2. Benchmarks: All of the above, as well as system metrics from node.js. For some it will be helpful to generate a flamegraph and heap snapshots.
3. Load Testing: All of the above. We will also get benefit out of tools like request tracing, diagnostics_channels, integrations with tools like NSolid and DataDog. Lastly we will want tools to diagnose client side discrepancies and networking issues if the server is falling over.

I am sure there is also more I am missing here. In my experience, the types of performance impacting issues which happen in the wild can be pretty convoluted and while our initial goal is to focus on the framework performance, I can see us finding some value in helping better document these things for our users when we are further along.

[bjohansebas]

i don’t have much experience measuring application performance, but the types of performance tests you’re proposing sound good to me. i would focus more on Benchmarks and Load Testing.

i don’t think we should put too much focus on Micro-benchmarks, although i know we’ll get hundreds of PRs claiming performance improvements in specific functions. So it wouldn’t hurt to have something to measure that

I can see us finding some value in helping better document these things for our users when we are further along.

100%, for heavy workloads, things like offloading to a new worker (worker_threads) or using a new process (child_process.fork) so that Node.js doesn't get blocked, that can definitely help users

[wesleytodd]

i don’t think we should put too much focus on Micro-benchmarks, although i know we’ll get hundreds of PRs claiming performance improvements in specific functions. So it wouldn’t hurt to have something to measure that

I agree on all points. The goal of giving guidance around micro-benchmarks would be to have a resource to point to which shows the types of benchmarks we prefer instead, to help guide the folks in those PRs.

100%, for heavy workloads, things like offloading to a new worker (worker_threads) or using a new process (child_process.fork) so that Node.js doesn't get blocked, that can definitely help users

Yeah, this kind of stuff would be awesome to build load tests around to show what types of workload benefit from this kind of setup and by how much. Sometimes you would be surprised at how seemingly minor details can break how you expect a system to behave.

[kjugi]

I have just noticed this issue after creating this one: #34. If you think it's duplicated or can be tackled in the same issue - we can close mine

What do you guys think about watching event loop?
Reference links:

• https://nodesource.com/blog/node-js-performance-monitoring-part-3-debugging-the-event-loop
• https://nodejs.org/id/learn/asynchronous-work/event-loop-timers-and-nexttick

[wesleytodd]

If you think it's duplicated or can be tackled in the same issue - we can close mine

@kjugi I think the scope is enough different that we can track them separately. This one is for what types of "tests" and that is "for those test types, what kinds of app use cases".

What do you guys think about watching event loop?

Yeah absolutely this is a very important metric. I believe that NodeSource and DataDog both track this, so we should be able to leverage their instrumentation for this. That should be included in our target comparison metrics for the http load tests for sure. Maybe the next step on this if folks agree is to start a markdown doc for this we can add details like this to?