ACLiC compilation confuses compiled binaries with shared objects, breaking compilation in some cases

[eguiraud]

• Checked for duplicates

Describe the bug

Building ROOT on Arch Linux and then running ctest -R "(dataframe|datasource)" -- -j8 results in several test failures.
There are various possible failure modes -- the command above typically results in these kind of errors:

1369: Processing /home/jalopezg/CERN/roottest/root/dataframe/test_snapshot.C+...
1369: Info in <TUnixSystem::ACLiC>: creating shared library /home/jalopezg/CERN/build/roottest/root/dataframe/test_snapshot_C.so
1369: /usr/bin/ld: /home/jalopezg/CERN/build/roottest/root/dataframe/test_readFcc: _ZSt4cout: invalid version 6 (max 0)
1369: /usr/bin/ld: /home/jalopezg/CERN/build/roottest/root/dataframe/test_readFcc: error adding symbols: bad value
1369: collect2: error: ld returned 1 exit status
1369: Error in <ACLiC>: Executing 'cd "/home/jalopezg/CERN/build/roottest/root/dataframe" ; c++ -fPIC -c -O3 -DNDEBUG -std=c++14 -Wno-implicit-fallthrough -Wno-noexcept-type -pipe -W -Woverloaded-virtual -fsigned-char -pthread  -I$ROOTSYS/include -I/home/jalopezg/CERN/build/roottest/root/dataframe -I"/home/jalopezg/CERN/build/etc/" -I"/home/jalopezg/CERN/build/etc//cling" -I"/home/jalopezg/CERN/build/include/" -I"/home/jalopezg/CERN/build/include" -I"/home/jalopezg/CERN/build/roottest/root/dataframe"   -D__ACLIC__ "/home/jalopezg/CERN/build/roottest/root/dataframe/test_snapshot_C_ACLiC_dict.cxx" ; c++ -O3 -DNDEBUG "/home/jalopezg/CERN/build/roottest/root/dataframe/test_snapshot_C_ACLiC_dict.o" -shared   "/usr/lib/libdl.so" "/usr/lib/libc.so" "/usr/lib/libgcc_s.so" "/usr/lib/libm.so" "/usr/lib/libstdc++.so" "/home/jalopezg/CERN/build/lib/libRint.so" "/home/jalopezg/CERN/build/lib/libCore.so" "/usr/lib/libpthread.so.0" "/usr/lib/libpcre.so.1" "/usr/lib/liblzma.so.5" "/usr/lib/liblz4.so.1" "/usr/lib/libz.so.1" "/usr/lib/libzstd.so.1" "/usr/lib/libnss_files.so.2" "/home/jalopezg/CERN/build/lib/libRIO.so" "/home/jalopezg/CERN/build/lib/libThread.so" "/home/jalopezg/CERN/build/lib/libCling.so" "/usr/lib/librt.so.1" "/usr/lib/libncursesw.so.6" "/usr/lib/ld-2.33.so" "/home/jalopezg/CERN/build/lib/libMathCore.so" "/home/jalopezg/CERN/build/lib/libImt.so" "/home/jalopezg/CERN/build/lib/libMultiProc.so" "/home/jalopezg/CERN/build/lib/libNet.so" "/usr/lib/libtbb.so.2" "/usr/lib/libssl.so.1.1" "/usr/lib/libcrypto.so.1.1" "/usr/lib/libnss_systemd.so.2" -o "/home/jalopezg/CERN/build/roottest/root/dataframe/test_snapshot_C.so" /home/jalopezg/CERN/build/lib/libROOTDataFrame.so /home/jalopezg/CERN/build/lib/libTreePlayer.so /home/jalopezg/CERN/build/lib/libTree.so /home/jalopezg/CERN/build/roottest/root/dataframe/test_readFcc /home/jalopezg/CERN/build/roottest/root/dataframe/par' failed!
1369: terminate called after throwing an instance of 'std::logic_error'
1369:   what():  basic_string::_M_construct null not valid

I did not manage to reproduce the problem outside of ctest and without running multiple tests concurrently -- it also seems that one of the tests "has" to be roottest-root-dataframe-test_snapshot_manytasks.

To Reproduce

The following repository provides a Dockerfile to reproduce the issue in a container https://gitlab.cern.ch/eguiraud/arch_aclic_bug .

Setup

ROOT master (but 6.22 also has issues) and arch linux.

I tried building from source (6.22 and master) as well as installing from the system repos -- failure modes are different but none can run all tests without problems.

Gentoo seems to also be affected.

[eguiraud]

i think there is a bug in how the aclic compilation command is built

i see this compilation command for one of the RDF roottests:

c++ -O3 -DNDEBUG "/home/blue/ROOT/master/_build/roottest/root/dataframe/writeFcc_C_ACLiC_dict.o" -shared   "/usr/lib/libdl.so" "/usr/lib/libc.so" "/usr/lib/libgcc_s.so" "/usr/lib/libm.so" "/usr/lib/libstdc++.so" "/home/blue/ROOT/master/_build/lib/libRint.so" "/home/blue/ROOT/master/_build/lib/libCore.so" "/usr/lib/libpthread.so.0" "/usr/lib/libpcre.so.1" "/usr/lib/liblzma.so.5" "/usr/lib/libxxhash.so.0" "/usr/lib/liblz4.so.1" "/usr/lib/libz.so.1" "/usr/lib/libzstd.so.1" "/usr/lib/libnss_files.so.2" "/home/blue/ROOT/master/_build/lib/libRIO.so" "/home/blue/ROOT/master/_build/lib/libThread.so" "/home/blue/ROOT/master/_build/lib/libCling.so" "/usr/lib/librt.so.1" "/usr/lib/libncursesw.so.6" "/usr/lib/ld-2.33.so" "/home/blue/ROOT/master/_build/lib/libMathCore.so" "/home/blue/ROOT/master/_build/lib/libImt.so" "/home/blue/ROOT/master/_build/lib/libMultiProc.so" "/home/blue/ROOT/master/_build/lib/libNet.so" "/usr/lib/libtbb.so.2" "/usr/lib/libssl.so.1.1" "/usr/lib/libcrypto.so.1.1" "/usr/lib/libnss_systemd.so.2" "/usr/lib/libcrypt.so.2" "/usr/lib/libp11-kit.so.0" "/usr/lib/libffi.so.7" -o "/home/blue/ROOT/master/_build/roottest/root/dataframe/writeFcc_C.so" /home/blue/ROOT/master/_build/lib/libROOTDataFrame.so /home/blue/ROOT/master/_build/lib/libTreePlayer.so /home/blue/ROOT/master/_build/lib/libTree.so /home/blue/ROOT/master/_build/roottest/root/dataframe/branchoverwrite /home/blue/ROOT/master/_build/roottest/root/dataframe/test_readerarray

i.e.

c++ <bunch of irrelevant stuff> -o "/home/blue/ROOT/master/_build/roottest/root/dataframe/writeFcc_C.so" /home/blue/ROOT/master/_build/lib/libROOTDataFrame.so /home/blue/ROOT/master/_build/lib/libTreePlayer.so /home/blue/ROOT/master/_build/lib/libTree.so /home/blue/ROOT/master/_build/roottest/root/dataframe/branchoverwrite /home/blue/ROOT/master/_build/roottest/root/dataframe/test_readerarray

The last two arguments are bogus, /home/blue/ROOT/master/_build/roottest/root/dataframe/branchoverwrite should not be there nor /home/blue/ROOT/master/_build/roottest/root/dataframe/test_readerarray

SImilarly for the failure reported in the issue description: there are 2 weird extra arguments appended to the compiler invocation that end up causing obscure aclic compilation failures.

[eguiraud]

At TSystem.cxx:3340, in TSystem::CompileMacro, the following call:

gInterpreter->GetSharedLibDeps("/home/blue/ROOT/master/roottest/root/dataframe/writeFcc_C.so", /*tryDyld*/ true);

returns

"writeFcc_C.so libROOTDataFrame.so libTreePlayer.so libTree.so branchoverwrite test_readerarray "

where I'm pretty sure the last two argument shouldn't be present.

[eguiraud]

In TInterpreter::GetSharedLibDeps, at some point we call interp->getDynamicLibraryManager()->searchLibrariesForSymbol("__libc_single_threaded@GLIBC_2.32", /*searchSystem*/false) and that returns "/home/blue/ROOT/master/_build/roottest/root/dataframe/branchoverwrite" in this broken usecase.

As branchoverwrite is a completely unrelated test, the linking of which actually breaks the ACLiC compilation of this macro, I guess we don't want to pick "__libc_single_threaded@GLIBC_2.32" from "/home/blue/ROOT/master/_build/roottest/root/dataframe/branchoverwrite"

[eguiraud]

Assigning to @vgvassilev as the problem seems to be that interp->getDynamicLibraryManager()->searchLibrariesForSymbol picks up unrelated executables as if they were libraries providing certain symbols

[vgvassilev]

@eguiraud, thanks for the detailed analysis. Unfortunately this works as designed and I am not sure if we can fix that behavior in a way that is reasonable when running the tests while keeping it working for the rest.

If the issue is just for the test suite, I'd propose slitting these tests into subfolders.

[pcanal]

@vgvassilev I am confused, why does getDynamicLibraryManager returns the names of executables? What is the purpose and/or why would an executable be used to provide symbols for other executables?

[Axel-Naumann]

Unfortunately this works as designed

That would mean the design is broken and needs to be reconsidered, possibly by re-introducing rootmap files. @pcanal FYI