[SUREFIRE-2041] Order test classes according to -Dtest

[jira-importer]

Aslak Hellesøy opened SUREFIRE-2041 and commented

I'm working on a Test Case Prioritization tool that uses machine learning to predict what tests are most likely to fail (based on historic changes and test results). The output from this tool is a list of test names, ordered by decreasing probability of failure. This tool can provide the order as test classes or as test methods. For example:

Class order:

• testing.Y
• testing.X

Method order:

• testing.Y#a
• testing.X#c
• testing.Y#b

Ideally, it would be possible to launch surefire with a test order that uses a list of tests to indicate what order they should be run in, e.g:

# class order
mvn test -Dsurefire.runOrder=test -Dtest=testing.Y,testing.X

# method order
mvn test -Dsurefire.runOrder=test -Dtest=testing.Y#a,testing.X#c,testing.Y#b

From what I can tell, the current design of Surefire wouldn't allow the kind of method ordering illustrated above. It cannot run {}testing.Y#a{}, then {}testing.X#c{}, then {}testing.Y#b{}. There is an inherent assumptions in TestsToRun that the overall ordering is by class.

I would therefore be ok with just class ordering for now.

Previous relates issues and pull requests:

• https://issues.apache.org/jira/browse/SUREFIRE-1405
• [SUREFIRE-1405] Allows user to extend RunOrder & RunOrderCalculator #169
• Enabling Surefire to run test classes and test methods in any order specified by a new runOrder #348

---

Remote Links:

• GitHub Pull Request #495
  
• GitHub Pull Request #560