The filter "tests" truncates the provided value at the first closing parenthesis.

[east1k]

Hello everyone!

The filter "tests" truncates the provided value at the first closing parenthesis, which can lead to the following:

• It breaks further use of the filter.
• It returns more results than expected.
• It returns fewer results than expected.

The bug appeared when we upgraded nunit.consolerunner.netcore from version 3.18.3 to version 3.20.0.
I suspect that the bug itself was introduced in version 3.19.0 as a result of this commit: 1af35ed.

The bug reproduces in the following cases:

• If a regex with parentheses is used, the parser may truncate the regex without an explicit error.

  Example

  If I run the following command:

  nunit "test.dll" -where "test =~ '(namespace1|namespace2)\\.test1'" --explore
  

  The parser will truncate to (namespace1|namespace2), and I will get all tests from these namespaces instead of namespace1.test1 and namespace2.test1.

• If a regex without parentheses is used but with test arguments, the parser may truncate the regex without an explicit error.

  Example

  If I run the following command:

  nunit "test.dll" -where "test =~ 'namespace\\.test1\\(1\\)|namespace\\.test2\\(2\\)'" --explore
  

  The parser will truncate to namespace\\.test1\\(1\\), and I will get only one test instead of two.

• If a regex with parentheses is used, it may break the regex with an error.

  Example

  If I run the following command:

  nunit "test.dll" -where "test =~ '(namespace\\.test1\\(1\\)|namespace\\.test2\\(2\\))'" --explore
  

  The parser will truncate to (namespace\\.test1\\(1\\), resulting in an invalid regex that will fail when creating Regex:

  Unit.Engine.NUnitEngineException : An exception occurred in the driver while exploring tests.
    ----> System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
    ----> System.Text.RegularExpressions.RegexParseException : Invalid pattern '(namespace\.test1\(1\)' at offset 22. Not enough )'s.
  
  --NUnitEngineException
  An exception occurred in the driver while exploring tests.
     at NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter) in D:\a\nunit-console\nunit-console\src\NUnitEngine\nunit.engine.core\Runners\DirectTestRunner.cs:line 81
     at NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter) in D:\a\nunit-console\nunit-console\src\NUnitEngine\nunit.engine\Runners\MasterTestRunner.cs:line 216
     at NUnit.ConsoleRunner.ConsoleRunner.ExploreTests(TestPackage package, TestFilter filter) in D:\a\nunit-console\nunit-console\src\NUnitConsole\nunit3-console\ConsoleRunner.cs:line 171
     at NUnit.ConsoleRunner.ConsoleRunner.Execute() in D:\a\nunit-console\nunit-console\src\NUnitConsole\nunit3-console\ConsoleRunner.cs:line 153
     at NUnit.ConsoleRunner.Program.Main(String[] args) in D:\a\nunit-console\nunit-console\src\NUnitConsole\nunit3-console\Program.cs:line 117
  --
  TargetInvocationException
  Exception has been thrown by the target of an invocation.
     at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
     at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
     at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
     at NUnit.Engine.Drivers.NUnitNetCore31Driver.ExecuteMethod(MethodInfo method, Object[] args) in D:\a\nunit-console\nunit-console\src\NUnitEngine\nunit.engine.core\Drivers\NUnitNetCore31Driver.cs:line 207
     at NUnit.Engine.Drivers.NUnitNetCore31Driver.ExecuteMethod(String methodName, Object[] args) in D:\a\nunit-console\nunit-console\src\NUnitEngine\nunit.engine.core\Drivers\NUnitNetCore31Driver.cs:line 189
     at NUnit.Engine.Drivers.NUnitNetCore31Driver.Explore(String filter) in D:\a\nunit-console\nunit-console\src\NUnitEngine\nunit.engine.core\Drivers\NUnitNetCore31Driver.cs:line 164
     at NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter) in D:\a\nunit-console\nunit-console\src\NUnitEngine\nunit.engine.core\Runners\DirectTestRunner.cs:line 77
  --
  RegexParseException
  Invalid pattern '(namespace\.test1\(1\)' at offset 22. Not enough )'s.
     at System.Text.RegularExpressions.RegexParser.ScanRegex()
     at System.Text.RegularExpressions.RegexParser.Parse(String pattern, RegexOptions options, CultureInfo culture)
     at System.Text.RegularExpressions.Regex.Init(String pattern, RegexOptions options, TimeSpan matchTimeout, CultureInfo& culture)
     at System.Text.RegularExpressions.Regex..ctor(String pattern, CultureInfo culture)
     at System.Text.RegularExpressions.Regex..ctor(String pattern)
     at NUnit.Framework.Internal.Filters.ValueMatchFilter.Match(String input)
     at NUnit.Framework.Internal.Filters.FullNameFilter.Match(ITest test)
     at NUnit.Framework.Internal.TestFilter.Pass(ITest test, Boolean negated)
     at NUnit.Framework.Internal.TestFilter.Pass(ITest test)
     at NUnit.Framework.Internal.TestSuite..ctor(TestSuite suite, ITestFilter filter)
     at NUnit.Framework.Internal.TestAssembly..ctor(TestAssembly assembly, ITestFilter filter)
     at NUnit.Framework.Api.NUnitTestAssemblyRunner.ExploreTests(ITestFilter filter)
     at NUnit.Framework.Api.FrameworkController.ExploreTests(String filter)
     at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
     at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
  
  

• If a TestFixture with constructor arguments is used, the parser will truncate the value to TestFixture.

  Example

  If I run the following command:

  nunit "test.dll" -where "test == 'namespace.class(1).test1(1)'" --explore
  

  The parser will truncate to namespace.class(1), and I will get all tests from the class instead of a specific test.

Test cases for TestSelectionParserTests.cs to reproduce the issue:

new TestCaseData("test == namespace.class(1).test1(1)", "<test>namespace.class(1).test1(1)</test>"),
new TestCaseData("test == \"namespace.class(1).test1(1)\"", "<test>namespace.class(1).test1(1)</test>"),
new TestCaseData("test == 'namespace.class(1).test1(1)'", "<test>namespace.class(1).test1(1)</test>"),
new TestCaseData("test =~ \"(namespace\\.test1\\(1\\)|namespace\\.test2\\(2\\))\"", "<test re='1'>(namespace.test1(1)|namespace.test2(2))</test>"),
new TestCaseData("test =~ '(namespace\\.test1\\(1\\)|namespace\\.test2\\(2\\))'", "<test re='1'>(namespace.test1(1)|namespace.test2(2))</test>"),
new TestCaseData("test =~ /(namespace\\.test1\\(1\\)|namespace\\.test2\\(2\\))/", "<test re='1'>(namespace.test1(1)|namespace.test2(2))</test>"),
new TestCaseData("test =~ \"(namespace1|namespace2)\\.test1\"", "<test re='1'>(namespace1|namespace2).test1</test>"),
new TestCaseData("test =~ '(namespace1|namespace2)\\.test1'", "<test re='1'>(namespace1|namespace2).test1</test>"),
new TestCaseData("test =~ /(namespace1|namespace2)\\.test1/", "<test re='1'>(namespace1|namespace2).test1</test>"),

[east1k]

To be honest, I don't quite understand the purpose of the GetTestName method, so I'm not sure I can implement the necessary fix.
One of its tasks is to normalize the test name by removing extra spaces, but I would leave this up to the user, as it was done before.
But if this is important for some users, I tried to combine the old and new behavior in the attached pull request: #1681

Changes:

• If the filter is "tests" and the operator is regex, take the value without normalization.
• If the filter is "tests" and the operator is comparison, take the value using "GetTestName".
• Function "GetTestName" takes the value to the end, rather than truncating at the first closing parenthesis.
• Function "GetTestName" removes extra spaces everywhere except in quoted arguments.
• Function "GetTestName" ignores escaped quotes.

[CharliePoole]

Thanks for your work on this. Oddly enough, I've been running into the same problem myself, with regard to parameterized fixtures, and haven't had time to look into it. I'll check to see if your PR helps in that situation as well.

Re your comment about the purpose of the GetTestName method...

The user has several ways to control the names of tests...

1. By naming classes and methods within the test code itself. In that case, the NUnit framework decides what the name of the test should be using a standard naming template.
2. In addition, the user may provide an alternate naming template, for use by the framework.
3. Finally, the user may directly set the test name using a property, e.g. the TestName property of TestCaseAttribute.

GetTestName has to interpret the command-line entry in light of all those possibilities, which makes it a bit confusing. It should probably be commented to indicate why it does various things. I'll take a look. As far as removing multiple spaces, I'm not sure about that, so I'll look into it further.

UPDATE: GetTestName is intending to remove spaces only between arguments, not within arguments, because NUnit names tests like MethodName(arg1,arg2) without extraneous spaces. Eliminating them allows for the user possibly introducing extraneous spaces on the command-line. It's not supposed to remove spaces within a single (possibly quoted) argument. If it's doing that, then it's definitely a bug.

[CharliePoole]

This issue has been resolved in version 3.20.1

The release is available on:
GitHub.
NuGet packages are also available NuGet.org and
Chocolatey Packages may be found at Chocolatey.org