Feature request: Support snapshot testing

[fisker]

We test rules with a custom rule tester doing a snapshot test.

The built-in RuleTester can't customize the output assertion, we don't have the output to send to RuleTester, so we have to use SourceCodeFixer to apply fix from messages. Full implementation

We definitely want a better way to do snapshot test, maybe pass an assert option in future, but before that, can we expose SourceCodeFixer, so I can apply fix from the message?

The version of ESLint you are using.

8.0.0-beta.0

The problem you want to solve.

Fix our rule tester.

Are you willing to submit a pull request to implement this change?

Yes.

[nzakas]

Can you provide an example of the type of test that you would like to write?

[fisker]

The tests

The snapshots

[nzakas]

Sorry, what I meant was, can you show how you are currently using SourceCodeFixer? My preference is not to expose that as an API, but I’m curious if the pattern you were using is something we should just build into RuleTester. Please provide as much detail as possible to help us evaluate this.

[fisker]

Here is the line I use it, I've linked in the description.

Update: I'm sorry that I forgot to mention this is request to assert output for suggestions.

[nzakas]

Ah! I understand now, thanks for explaining. So you are wanting to make sure that each suggestion, when applied, is outputting the correct result?

If so, that seems like something we should build into RuleTester to work the same way that we currently work for regular fixes. Would that address your use case?

[fisker]

So you are wanting to make sure that each suggestion, when applied, is outputting the correct result?

That's right.

Would that address your use case?

I'm afraid it won't.
There is another problem, I can't do snapshot testing with builtin RuleTester, the output (which I don't have) is required for invalid cases, see this part

eslint/lib/rule-tester/rule-tester.js

Lines 928 to 944 in 81c60f4

	if (hasOwnProperty(item, "output")) {
	if (item.output === null) {
	assert.strictEqual(
	result.output,
	item.code,
	"Expected no autofixes to be suggested"
	);
	} else {
	assert.strictEqual(result.output, item.output, "Output is incorrect.");
	}
	} else {
	assert.strictEqual(
	result.output,
	item.code,
	"The rule fixed the code. Please add 'output' property."
	);
	}

As mentioned in the issue description, unless we can customize the assertion step

maybe pass an assert option in future

[nzakas]

Hmm, okay. Well, I still think the correct solution here is to figure out how to modify RuleTester to support your case. It would be helpful if you could give an overview of how your custom-built system works, exactly how snapshots are generated, and what parts of RuleTester you are already using (or not).

And please be as detailed as you can.

[nzakas]

Note to self (mostly): You can currently test suggestions in RuleTester: https://eslint.org/docs/developer-guide/nodejs-api#testing-suggestions.

[lydell]

I use a hack to be able to use snapshot testing in output. My tests look like this:

https://github.com/lydell/eslint-plugin-simple-import-sort/blob/1f753fb3e5b8eadf6cf9fdda8790d01dd9eade26/test/imports.test.js#L78-L90

And the hack looks like so:

https://github.com/lydell/eslint-plugin-simple-import-sort/blob/1f753fb3e5b8eadf6cf9fdda8790d01dd9eade26/test/helpers.js#L6-L22

There are some things very specific to that plugin, but the gist is that it replace assert.strictEqual with my own function, that tries to detect the RuleTester and do something else in that case.

[nzakas]

@lydell thanks, that’s helpful. I think it’s clear people want snapshot testing, so I’ll ask you the same question: what would we need to add to RuleTester that would eliminate your hacks?