Extensions are no longer loading as is currently documented

[mmalandra-kb4]

NUnit Engine/API versions 3.19.1.

How should extensions be loaded now? With the latest nunit-console (3.19.1) and engines/api, our extensions are no longer loading.

From the source code, you can see a major change here where the code is only looking at specific folders that match hardcoded patterns:

public void FindExtensionAssemblies(Assembly hostAssembly)
{
    log.Info($"FindExtensionAssemblies called for host {hostAssembly.FullName}");

    bool isChocolateyPackage = System.IO.File.Exists(Path.Combine(hostAssembly.Location, "VERIFICATION.txt"));
    string[] extensionPatterns = isChocolateyPackage
        ? new[] { "nunit-extension-*/**/tools/", "nunit-extension-*/**/tools/*/" }
        : new[] { "NUnit.Extension.*/**/tools/", "NUnit.Extension.*/**/tools/*/" };


    IDirectory startDir = _fileSystem.GetDirectory(AssemblyHelper.GetDirectoryName(hostAssembly));

    while (startDir != null)
    {
        foreach (var pattern in extensionPatterns)
            foreach (var dir in _directoryFinder.GetDirectories(startDir, pattern))
                ProcessDirectory(dir, true);

        startDir = startDir.Parent;
    }
}

Does the documentation around loading extensions need to be updated, or is there a way to get engine to allow for other folders to contain extensions?

[mazharenko]

Is Chocolatey involved in your case? Then have a look at #1614

Addins files must still be supported #1504

[mmalandra-kb4]

We are not using chocolately.

[CharliePoole]

@mmalandra-kb4

This change is by design. You can review the discussion on #488 and #1504 for some of the reasons.

Have you used the new --extensionsDirectory option? If you have and it's not working correctly, please provide details of your command-line and directory structure.

[mmalandra-kb4]

I have tried both the extensionDirectory option and the NUNIT_EXTENSION_DIRECTORY environment variable, both of which do not seem to work.

Here is an example docker file that I'm testing this on:

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS builder
ARG ROOT_DIR=src
ARG BUILD_DIR=/var/code/$ROOT_DIR
ARG PROJECT_DIR=$BUILD_DIR/TestProject1

COPY ./$ROOT_DIR/ $BUILD_DIR

RUN cd $PROJECT_DIR && dotnet publish -c Release -o $BUILD_DIR/output -f net8.0 -r linux-arm64 --no-self-contained
RUN dotnet tool install --global nunit.consolerunner.netcore  --version 3.19.1

FROM mcr.microsoft.com/dotnet/runtime:8.0 AS runner
ARG ROOT_DIR=src
ARG BUILD_DIR=/var/code/$ROOT_DIR
ENV TEST_DLL=/var/code/TestProject1.dll

COPY --from=builder /root/.dotnet/tools /root/.dotnet/tools
COPY --from=builder /root/.dotnet/tools/.store /root/.dotnet/tools/.store
COPY --from=builder $BUILD_DIR/output/*.dll /var/code/
COPY --from=builder $BUILD_DIR/output/*.dll /var/code/NUnit.Extension.Test/test/tools/
ENV PATH="$PATH:/root/.dotnet/tools"


RUN NUNIT_PATH=$(which nunit) && ln -sf $NUNIT_PATH /usr/local/bin/nunit3-console

WORKDIR /var/code
CMD nunit3-console $TEST_DLL  --extensionDirectory=/var/code --list-extensions --trace=Info 2>&1 | tee test.log

The actual unit test is small in my sample project:

using System.Diagnostics;
using System.IO;
using NUnit.Framework;

namespace TestProject1;

public class Tests
{
    [OneTimeSetUp]
    public void StartTest()
    {
        Trace.Listeners.Add(new ConsoleTraceListener());
    }

    [OneTimeTearDown]
    public void EndTest()
    {
        Trace.Flush();
    }

    [Test]
    public void Test1()
    {
        var startDir = "/var/code/";
        var pattern = "NUnit.Extension.Test";
        var dirs = Directory.GetDirectories(startDir, pattern);
        TestContext.Progress.WriteLine(dirs.Length);
        Assert.That(() => 1 == dirs.Length);
    }
}

And the extension is here:

using System.IO;

namespace TestProject1;

[NUnit.Engine.Extensibility.Extension]
public class MyListener : NUnit.Engine.ITestEventListener
{
    public void OnTestEvent(string report)
    {
        File.WriteAllText("/tmp/MY_REPORT.txt", report);
    }
}

My docker-compose.yml looks like:

name: nunit-test
services:
  nunit-test:
    platform: linux/arm64
    build:
      context: .
      dockerfile: ./local.Dockerfile

To run the test, I am running docker compose up --build

The output from the run:

nunit-test-1  | NUnit NetCore Console Runner 3.19.1 (Release)
nunit-test-1  | Copyright (c) 2022 Charlie Poole, Rob Prouse
nunit-test-1  | Tuesday, 04 February 2025 20:52:06
nunit-test-1  | 
nunit-test-1  | Runtime Environment
nunit-test-1  |    OS Version: Debian GNU/Linux 12 (bookworm)
nunit-test-1  |   Runtime: .NET 8.0.12
nunit-test-1  | 
nunit-test-1  | User Extension Directories
nunit-test-1  |   /var/code
nunit-test-1  | 
nunit-test-1  | Installed Extensions
nunit-test-1  |   Extension Point: /NUnit/Engine/NUnitV2Driver
nunit-test-1  |   Extension Point: /NUnit/Engine/TypeExtensions/IService
nunit-test-1  |   Extension Point: /NUnit/Engine/TypeExtensions/ITestEventListener
nunit-test-1  |   Extension Point: /NUnit/Engine/TypeExtensions/IDriverFactory
nunit-test-1  |   Extension Point: /NUnit/Engine/TypeExtensions/IProjectLoader
nunit-test-1  |   Extension Point: /NUnit/Engine/TypeExtensions/IResultWriter
nunit-test-1  | 
nunit-test-1  | Test Files
nunit-test-1  |     /var/code/TestProject1.dll
nunit-test-1  | 
nunit-test-1  | 1
nunit-test-1  | 
nunit-test-1  | Run Settings
nunit-test-1  |     DisposeRunners: True
nunit-test-1  |     InternalTraceLevel: Info
nunit-test-1  |     WorkDirectory: /var/code
nunit-test-1  |     NumberOfTestWorkers: 10
nunit-test-1  | 
nunit-test-1  | Test Run Summary
nunit-test-1  |   Overall result: Passed
nunit-test-1  |   Test Count: 1, Passed: 1, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
nunit-test-1  |   Start time: 2025-02-04 20:52:06Z
nunit-test-1  |     End time: 2025-02-04 20:52:06Z
nunit-test-1  |     Duration: 0.079 seconds
nunit-test-1  | 
nunit-test-1  | Results (nunit3) saved as TestResult.xml
nunit-test-1 exited with code 0

[CharliePoole]

In addition to fixing this, I'd like to add a package test to ensure that it keeps working.

[CharliePoole]

@mmalandra-kb4 I don't see that you have a .addins file anywhere. Do you?

[CharliePoole]

I am confirming this. The error is both obvious (once you see it) and subtle.

Neither of the methods that locate candidate assemblies actually search for extensions in those assemblies. It's done on a just-in-time basis when a call is made to access something that requires initialization to be complete, for example, a list of all extensions.

To avoid unnecessary searching for extensions we check to see if initializaton is complete. Unfortunately, at the point where the ConsoleRunner examines the extra directories you provide, initialization is already complete, so nothing further is done.

Introducing the --extensionDirectory means that we can no longer rely on loading of extensions happening all at once. We must now support partial initializaton with subsequent addition of new extensions. This means a bit of a rewrite of the initialization logic in the long run.

However, in the special case of the use of this option with the runner, initialization has only just completed and no extensions are yet in use. I'll check to see if a simpler immediate fix is possible.

[CharliePoole]

@mmalandra-kb4

From my manual tests, I believe this is fixed. At the moment, we can't adequately test it in our CI build, which needs to be enhanced to deal with extensions stored in alternate directories. For that reason, I'd appreciate your verifying it. The fix is available on our myget feed as version 3.19.3-alpha.1.

In case it's not clear in the help, please note that we do not look for assemblies in the directory you specify, but only for .addins files. If an entry points to some other directory, we will look at additional addins files if present, otherwise at assemblies.

This is not a change from how it worked in the past, but it sometimes confuses people. In fact, I'm wondering whether to modify it so that the initial directory works in the same way as subsequent directories. What are your thoughts?