Skip to content
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ By default, cucumber-js runs the entire suite and reports all the failures. This
## Retry failing tests

Use `--retry <int>` to rerun tests that have been failing. This can be very helpful for flaky tests.
To only retry failing tests in a subset of test use `--retryTagFilter <EXPRESSION>` (use the same as in Use [Tags](#tags))
To only retry failing tests in a subset of test use `--retry-tag-filter <EXPRESSION>` (use the same as in Use [Tags](#tags))

## Transpilation

Expand Down
44 changes: 32 additions & 12 deletions features/retry.feature
Original file line number Diff line number Diff line change
@@ -1,17 +1,37 @@
Feature: Retry flaky tests

Using the `--retry` flag will retry failing tests for the specified number of times
Additionally using the `--retryTagFilter` flag will re-run only tests matching the tag expression
Additionally using the `--retry-tag-filter` flag will re-run only tests matching the tag expression

@spawn
Scenario: running Cucumber JS with --retryTagFilter but no positive --retry will fail
When I run cucumber-js with `--retryTagFilter @flaky`
Scenario: running Cucumber JS with --retry-tag-filter but no positive --retry will fail
When I run cucumber-js with `--retry-tag-filter @flaky`
Then the error output contains the text:
"""
Error: a positive --retry count must be specified when setting --retryTagFilter
Error: a positive --retry count must be specified when setting --retry-tag-filter
"""
And it fails

Scenario: running Cucumber JS with --retryTagFilter in camel case will result in a warning
Given a file named "features/a.feature" with:
"""
Feature:
Scenario:
Given a step
"""
Given a file named "features/step_definitions/cucumber_steps.js" with:
"""
const {Given} = require('cucumber')

Given(/^a step$/, function() {})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter @flaky`
Then the error output contains the text:
"""
the argument --retryTagFilter is deprecated and will be removed in a future release; please use --retry-tag-filter
"""
But it passes

Scenario: running Cucumber JS with negative --retry will fail
When I run cucumber-js with `--retry -1`
Then the error output contains the text:
Expand Down Expand Up @@ -322,7 +342,7 @@ Feature: Retry flaky tests
"""
And it fails

Scenario: retrying a flaky test matching --retryTagFilter will eventually make it pass
Scenario: retrying a flaky test matching --retry-tag-filter will eventually make it pass
Given a file named "features/a.feature" with:
"""
Feature:
Expand All @@ -344,12 +364,12 @@ Feature: Retry flaky tests
throw 'fail'
})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter '@flaky'`
When I run cucumber-js with `--retry 1 --retry-tag-filter '@flaky'`
Then scenario "Flaky" attempt 0 step "Given a flaky step" has status "failed"
Then scenario "Flaky" attempt 1 step "Given a flaky step" has status "passed"
And it passes

Scenario: a flaky test not matching --retryTagFilter won't re-run and just fail
Scenario: a flaky test not matching --retry-tag-filter won't re-run and just fail
Given a file named "features/a.feature" with:
"""
Feature:
Expand All @@ -371,11 +391,11 @@ Feature: Retry flaky tests
throw 'fail'
})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter '@not_flaky'`
When I run cucumber-js with `--retry 1 --retry-tag-filter '@not_flaky'`
Then scenario "Flaky" step "Given a flaky step" has status "failed"
And it fails

Scenario: retrying a flaky test matching --retryTagFilter will eventually make it pass but not-matching will not be retried (AND operator between tags)
Scenario: retrying a flaky test matching --retry-tag-filter will eventually make it pass but not-matching will not be retried (AND operator between tags)
Given a file named "features/a.feature" with:
"""
Feature:
Expand Down Expand Up @@ -410,13 +430,13 @@ Feature: Retry flaky tests
throw 'fail'
})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter '@flaky and @anOtherTag'`
When I run cucumber-js with `--retry 1 --retry-tag-filter '@flaky and @anOtherTag'`
Then scenario "Flaky" attempt 0 step "Given a flaky step" has status "failed"
Then scenario "Flaky" attempt 1 step "Given a flaky step" has status "passed"
And scenario "Also Flaky" step "Given an other flaky step" has status "failed"
And it fails

Scenario: retrying a flaky test matching --retryTagFilter will eventually make it pass but not-matching will not be retried (OR operator between tags)
Scenario: retrying a flaky test matching --retry-tag-filter will eventually make it pass but not-matching will not be retried (OR operator between tags)
Given a file named "features/a.feature" with:
"""
Feature:
Expand Down Expand Up @@ -464,7 +484,7 @@ Feature: Retry flaky tests
throw 'fail'
})
"""
When I run cucumber-js with `--retry 1 --retryTagFilter '@anOtherTag or @oneMoreTag'`
When I run cucumber-js with `--retry 1 --retry-tag-filter '@anOtherTag or @oneMoreTag'`
Then scenario "Flaky" attempt 0 step "Given a flaky step" has status "failed"
And scenario "Flaky" attempt 1 step "Given a flaky step" has status "passed"
And scenario "Also Flaky" attempt 0 step "Given an other flaky step" has status "failed"
Expand Down
14 changes: 10 additions & 4 deletions features/support/world.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,25 +60,31 @@ export class World {
})
} else {
const stdout = new PassThrough()
const stderr = new PassThrough()
const cli = new Cli({
argv: args,
cwd,
stdout,
stderr,
})
let error: any, stderr: string
let error: any
try {
const { success } = await cli.run()
if (!success) {
error = new Error('CLI exited with non-zero')
error.code = 42
}
stderr = ''
} catch (err) {
error = err
stderr = VError.fullStack(error)
stderr.write(VError.fullStack(error))
}
stdout.end()
result = { error, stdout: await toString(stdout), stderr }
stderr.end()
result = {
error,
stdout: await toString(stdout),
stderr: await toString(stderr),
}
}
const envelopes: messages.Envelope[] = []
const messageOutputPath = path.join(cwd, messageFilename)
Expand Down
4 changes: 2 additions & 2 deletions src/cli/argv_parser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ const ArgvParser = {
validateRetryOptions(options: IParsedArgvOptions): void {
if (options.retryTagFilter !== '' && options.retry === 0) {
throw new Error(
'a positive --retry count must be specified when setting --retryTagFilter'
'a positive --retry count must be specified when setting --retry-tag-filter'
)
}
},
Expand Down Expand Up @@ -188,7 +188,7 @@ const ArgvParser = {
0
)
.option(
'--retryTagFilter <EXPRESSION>',
'--retryTagFilter, --retry-tag-filter <EXPRESSION>',
`only retries the features or scenarios with tags matching the expression (repeatable).
This option requires '--retry' to be specified.`,
ArgvParser.mergeTags,
Expand Down
26 changes: 22 additions & 4 deletions src/cli/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import { doesNotHaveValue } from '../value_checker'
import Gherkin from 'gherkin'
import { ISupportCodeLibrary } from '../support_code_library_builder/types'
import { IParsedArgvFormatOptions } from './argv_parser'
import { Console } from 'console'

const { incrementing, uuid } = IdGenerator

Expand All @@ -42,16 +43,33 @@ export default class Cli {
private readonly argv: string[]
private readonly cwd: string
private readonly stdout: IFormatterStream
private readonly console: Console

constructor({ argv, cwd, stdout }) {
constructor({ argv, cwd, stdout, stderr }) {
this.argv = argv
this.cwd = cwd
this.stdout = stdout
this.console = new Console(stderr)
}

async getConfiguration(): Promise<IConfiguration> {
const fullArgv = await getExpandedArgv({ argv: this.argv, cwd: this.cwd })
return ConfigurationBuilder.build({ argv: fullArgv, cwd: this.cwd })
const fullArgv = await getExpandedArgv({
argv: this.argv,
cwd: this.cwd,
})
this.lintArgv(fullArgv)
return ConfigurationBuilder.build({
argv: fullArgv,
cwd: this.cwd,
})
}

private lintArgv(fullArgv: string[]): void {
if (fullArgv.includes('--retryTagFilter')) {
this.console.warn(
'the argument --retryTagFilter is deprecated and will be removed in a future release; please use --retry-tag-filter'
)
}
}

async initializeFormatters({
Expand Down Expand Up @@ -83,7 +101,7 @@ export default class Cli {
}
if (type === 'progress-bar' && !(stream as TtyWriteStream).isTTY) {
const outputToName = outputTo === '' ? 'stdout' : outputTo
console.warn(
this.console.warn(
`Cannot use 'progress-bar' formatter for output to '${outputToName}' as not a TTY. Switching to 'progress' formatter.`
)
type = 'progress'
Expand Down
1 change: 1 addition & 0 deletions src/cli/run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ export default async function run(): Promise<void> {
argv: process.argv,
cwd,
stdout: process.stdout,
stderr: process.stderr,
})

let result: ICliRunResult
Expand Down