[Feature] Allow users to define ordering for TestNG listeners

[krmahadevan]

As a user, I should be allowed to determine the order in which TestNG listeners will be invoked in my project (irrespective of whether those listeners were authored by me or if they were just getting resolved via some dependency).

TestNG currently has no way of ensuring that listeners are invoked in any specific order. The only round about approach that exists today is to build a container listener which would internally get all the listeners that a user wants to be registered within it and then it internally orchestrates execution of those listeners in a user determined fashion.

But this approach has a lot of pitfalls. Users have no control on listeners that were injected into TestNG via their classpath dependencies.

To mitigate all of this, TestNG should be able to do something like below:

1. User defines a comparator that basically takes in two listener objects (implementations of ITestNGListener)
2. Using the comparator the user determines the execution order.
3. User plugs in this comparator as a configuration (or) as a special listener. The special listener path if taken would mean that this listener can only have 1 instance per TestNG execution.
4. TestNG uses this new comparator to determine order and reverse-order for regular listeners and also for dealing with all reporters (including the TestNG default ones).
5. TestNG should NOT apply this comparator logic to special listeners (the notion of special listeners were introduced into TestNG as part of this commit and should be available in the next upcoming TestNG version (7.8.0 perhaps?)

For more context, refer to this discussion

[juherr]

Remarks:

• I think reporters should be excluded from this logic. They are not involved in the engine worflow (only read).
• What you call as specific listeners should be treated like any other listeners (except the internal ones which are now directly called) because they are just part of the default order logic.

[krmahadevan]

@juherr

What you call as specific listeners should be treated like any other listeners (except the internal ones which are now directly called) because they are just part of the default order logic.

What would be the expected behaviour when the user provided a comparator for ordering the listeners and still wanted some listeners to be excluded from the ordering and so specified them via the JVM argument ?

If we don't exclude them then we have broken that functionality.

[bobshie]

There is the other scenario, users define several listeners: AListener, BListener, CListener. maybe users want to invoke some afterconfig/afterinvocation as the reversed sequence with beforeconfig/afterinvocation.

for example: AListener.beforeconfig, BListener.beforeconfig, CListener.beforeconfig, CListener.afterconfig, BListener.afterconfig, AListener.afterconfig.

[juherr]

Could someone write a specification draft that will be used as documentation later?
It will allow us to discuss details on a concrete thing.

[krmahadevan]

Could someone write a specification draft that will be used as documentation later? It will allow us to discuss details on a concrete thing.

@juherr - I would like to pick this up and get this done. I have created a discussion #2916 Can we vet this out so that I can get started on this ?