doc: recommend node-stress-single-test for flaky tests

[trivikr]

Recommend node-stress-single-test in the collaborator guide and onboarding
docs for validating fixes to flaky tests.

Refs: https://openjs-foundation.slack.com/archives/C03BJP63CH0/p1782716150521499

[nodejs-github-bot]

Review requested:

• @nodejs/tsc

[sxa]

Suggest adding the hyperlink in, but approving regardless 👍🏻

[sxa]

Suggested change

	fixing a flaky test, it is recommended to run [`node-stress-single-test`][]
	fixing a flaky test, it is recommended to run [`node-stress-single-test`][https://ci.nodejs.org/job/node-stress-single-test]

[trivikr]

Mentioning the hyperlink in the bottom improves the source code readability, right?

It's done twice on line 289 for example.

[reliability repository][] and [guide on how to deal with CI flakes][].

[lpinca]

It is not very useful if the flaky test is in test/parallel because afaik the job runs the test sequentially.

[trivikr]

It is not very useful if the flaky test is in test/parallel because afaik the job runs the test sequentially.

The node-stress-single-test is still helpful for sequential tests.
Do you have a suggestion on how to phrase it in documentation?

[lpinca]

How about something like "When fixing a flaky test in test/sequentual, it is recommended to run node-stress-single-test. It is also good for tests in in test/parallel, but not very reliable because the job runs the test sequentially".

[sxa]

How about something like "When fixing a flaky test in test/sequentual, it is recommended to run node-stress-single-test. It is also good for tests in in test/parallel, but not very reliable because the job runs the test sequentially".

I'll be honest I've found it to be good for parallel tests as well. Even though it's theoretically possible that we'd get issues with bad interactions/conflicts when running alongside other tests in the parallel suite I don't think I've personally come across a situation where the fact that it's a parallel test has caused it not to show up in the stress job so I would dispute the "not very reliable" suggestion here and definitely would not discourage people from using the stress job for parallel tests. Curious as to whether others people's experiences are different from mine. And if a test is flaky because it's in the parallel suite, then the solution is probably to move it to sequential.

[lpinca]

I've run into a lot of cases where node-stress-single-test never failed and the flakiness was reproducible only by using -j 16 --repeat=1000.

And if a test is flaky because it's in the parallel suite, then the solution is probably to move it to sequential.

I disagree. In most cases it would only hide the underlying issue.

[sxa]

I've run into a lot of cases where node-stress-single-test never failed and the flakiness was reproducible only by using -j 16 --repeat=1000.

Thanks - good to know that I've just been lucky!

[joyeecheung]

I think there are multiple uses for the stress test:

1. For parallel testing, to put it under load
2. For sequential testing, to simply sample many many runs

It is not inherently less reliable when you run it sequentially, it just means in that case you are not putting it under load and are merely sampling many runs, which is also a valid, reliable way to use it, just be aware that it's for a different purpose.

One solution is to make -j configurable in Jenkins, I just added a workflow that allows stress testing with a configurable -j in #64118 for the same reason.