After upgrading to TestNG 7.5.0, setting ITestResult.status to `FAILURE` doesn't fail the test anymore

[asolntsev]

TestNG Version

• 7.5
• 7.6.0

Given

A project with Listener (my custom "soft assert" listener) which marks ITestResult status as FAILURE after the test.

Expected behavior

I expect my test to fail.

Actual behavior

After upgrading to TestNG 7.5, my test doesn't fail anymore. It's a bug.

Is the issue reproducible on runner?

• Maven
• Gradle
• IntelliJ

Test case sample

Please check this trivial project which stabely reproduces the problem:
https://github.com/asolntsev/testng-listeners-sample

[krmahadevan]

Can you please check in 7.6.0 which is the latest released version @asolntsev

[asolntsev]

@krmahadevan Just tried: the problem still reproduced with TestNG 7.6.0

[dr29bart]

is it a result of this fix: #2558 ?
TestNG 7.6.0

listeners = {ArrayList@5308}  size = 6
 0 = {ExitCodeListener@5311} 
 1 = {ConfigurationAwareTestNGReporter@5312} 
 2 = {TextReporter@5313} 
 3 = {JUnitXMLReporter@5314} 
 4 = {TestHTMLReporter@5315} 
 5 = {CustomSoftAssert@5316} 

TestNG 7.4.0

this.m_notifier.getTestListeners() = {ArrayList@5006}  size = 6
 0 = {CustomSoftAssert@5066} 
 1 = {TestHTMLReporter@5067} 
 2 = {JUnitXMLReporter@5068} 
 3 = {TextReporter@5069} 
 4 = {ConfigurationAwareTestNGReporter@5070} 
 5 = {ExitCodeListener@5071} 

[asolntsev]

@dr29bart Yes, it seems so. Generally, the idea to preserve the order of listeners is good. But still, TestNG own listeners should be executed AFTER all user-defined listeners (after test execution).

[hornyja4]

Hi guys, any progress in this? Totally agree with @asolntsev. This issue affected all our unit tests.