Isolated and Consistent Performance Testing in CI

[GroophyLifefor]

I propose a new approach for CI based performance testing using a dedicated Docker image that supports multi-version Node.js benchmarks, based on a template mechanism (e.g., autocannon, but pluggable).

This solution allows libraries and frameworks to:

• Run on CI via GitHub Actions and optionally post PR comments
• Run consistent performance tests in isolated environments
• Compare latest vs candidate commits using controlled templates
• Extract and analyze key metrics like req/sec, latency, and throughput
• Keep test dependencies isolated with per-test package.jsons

---

How It Works

Each test case is a subfolder under a expf-tests directory. ($TEST_DIR, default = "/expf-tests")
It includes:

expf-tests/
├── autocannon.mjs (Not a folder -> template)
├── my-test/
│   ├── run-test.mjs (Your test logic)
│   └── package.json (Your isolated deps)

A template file (default: autocannon.mjs) runs the test logic.

Then, inside the Docker container:

• Each folder is executed independently
• Metrics are collected via the chosen template
• Results are compared: latest vs candidate
• Final output is shown as Markdown, and optionally sent as a GitHub PR comment

Result Example: (whereas I put 1ms sleep in bodyParser.json() function)

[This comment is auto-generated by the perf runner]

## Performance Comparison for PR #1, Node.js 20

### Test Folder: json

| Metric              | Latest          | Candidate      | Difference       | Ratio       | Status        |
|---------------------|-----------------|----------------|------------------|-------------|---------------|
| Execution Time      | 10248.56 ms     | 10275.02 ms    | -26.46 ms        | ×1.00       | ❌ Regressed  |
| Average Latency     | 1.71 ms         | 9.34 ms        | -7.63 ms         | ×0.18       | ❌ Regressed  |
| Requests/Second     | 4556387.23 rps  | 989520.96 rps  | -3566866.27 rps  | ×0.22       | ❌ Regressed  |
| Errors              | 0               | 0              | 0                | N/A         | ✅ Unchanged  |

---

Pros

• Reproducible, consistent test runs across environments
• Isolates test dependencies (doesn’t pollute your repo's root package.json)
• Enables comparisons between commits or PRs
• Easily installable from Docker, CI-friendly
• Markdown reports & potential dashboard integration #28
• Clean developer experience with reusable templates
• Lightweight Dockerfile (~55MB, thanks to @mertssmnoglu)
• Multi-version Node.js test support (e.g., 16, 18, 20)

Repositories

Main repository: https://github.com/GroophyLifefor/expf
Testing purposed sample usage repository: https://github.com/GroophyLifefor/expf-lib
Real usage case example with body-parser: https://github.com/GroophyLifefor/body-parser

[GroophyLifefor]

After the last meeting, there was a general lack of activity, but steps can be taken to resolve this. If we clarify ourselves in this CI-based solution, I can move from my own repository to this repository and start implementing the necessary performance metrics. Then we can say that the first step is to move to this repository.

I believe that some issues are very important, such as maintaining public benchmarks. These parts are actually fragmented, and we can make more efficient progress if we divide up the work.

[wesleytodd]

I agree we need what you say in the title. What I am not sure I understand from the body of the issue is how the details here achieve that. I will ask a few question to hopefully help me understand:

Run on CI via GitHub Actions and optionally post PR comments

GHA runners are super inconsistent. We dont control them, but also they are multi-tenant. This means that some other job running on the same host can mess with the results of our tests. I remember discussing using Node Source infrastructure to run the tests we need consistency in. It is possible that was on a meeting and should be documented better, but @UlisesGascon let me know if that was not what you thought.

Compare latest vs candidate commits using controlled templates
Then, inside the Docker container....Results are compared: latest vs candidate

This is a great goal, but I think running both inside the same container is a no go. It suffers form the same "noisy neighbor" problem that the GHA runners do. If you did them in series it can cause other issues depending on the test being run (warm caches, werid filesystem things (what if the first tests writes logs to disk and fills it up, things like that).

I think we need to run them in independent (but still well controlled, containers work but so would consistent OS images) infra and then collect and compare the results outside of the container. FWIW, this is why I designed the runner the way I did. So lets maybe discuss options here that achieve the goals a bit better.

Extract and analyze key metrics like req/sec, latency, and throughput

Awesome, how do you see this overlapping with #19? One and the same? Or did you have a different direction in mind?

Keep test dependencies isolated with per-test package.jsons

I had been meaning to mention this. I think we need to put the package-lock.json in the results object for storage. The package.json alone does not have enough information.

Repositories

Would you be willing to start opening PRs to this repo? I follow too many and usually am pressed for time to review things, so it would help me (and maybe others?) a bunch if we tried to propose things as PRs so we can talk through details in a PR format.

[GroophyLifefor]

#38 (comment)

This comment is answered in 2025-08-06 meeting and to summarize the context,

• Since GHA is inconsistent, but docker image approach is a good way, we discussed about self-hosted runners or NodeSource based Infra.
• Since we use docker image as a OS, we can collect data what docker give us to access.
• I'll open a PR to integrate perf-runner to this repository, may it took few days based on my business.