Ensuring XUnitLogChecker is included in libraries outerloop #97135

carlossanlop · 2024-01-18T04:21:31Z

To decide whether to include the XUnitLogChecker.csproj among the projects to build to include in the helix payloads, check for TargetFrameworkIdentifier==.NETCoreApp instead of BuildTargetFramework==NetCoreAppCurrent.

TODO: Revert the temporary crash commit before merging.

ghost · 2024-01-18T04:21:37Z

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #96035

Still in draft because:

I am exploring adding XUnitLogChecker to clr.toolstests subset with many conditions, then adding that subset in the outerloop build runs. This is probably not the best approach but I need to test it.
I have temporary commits that need to be reverted before merging.

Here is an example of a libraries outerloop run where many tests failed due to not finding XUnitLogChecker.dll. My suspicion is that unlike the regular runtime libraries tests, outerloop does not retrieve the payloads that were built in separate preceding jobs. Here we just build in a previous step of the same job.
https://dev.azure.com/dnceng-public/public/_build/results?buildId=530217&view=logs&j=631ed29b-f0dd-5b15-38a0-c6c555556dea&t=b9941189-b5a8-5df3-6b3a-0964fabb3acd

Author:	carlossanlop
Assignees:	carlossanlop
Labels:	`area-Infrastructure-libraries`
Milestone:	-

carlossanlop · 2024-01-18T19:32:35Z

/azp run list

azure-pipelines · 2024-01-18T19:32:41Z

No pipelines are associated with this pull request.

carlossanlop · 2024-01-18T19:32:53Z

/azp list

carlossanlop · 2024-01-18T19:33:50Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-01-18T19:34:03Z

Azure Pipelines successfully started running 1 pipeline(s).

carlossanlop · 2024-02-14T01:39:18Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-02-14T01:39:26Z

Azure Pipelines successfully started running 1 pipeline(s).

carlossanlop · 2024-02-14T03:02:29Z

Got some progress here. XUnitLogChecker is being found now in outerloop, but it finds no dumps. Examples:

Queue: linux_musl-x64 Debug CoreCLR_Debug
Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=565516&view=logs&j=592eed89-78f5-5202-9da5-ad17df3bef4a&t=4feb010c-08b1-5201-8e2c-97e3caf24223&l=60
Log file: https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-c3e9c219d5d34b25aa/System.IO.Compression.Brotli.Tests/1/console.d33446c2.log?helixlogtype=result
Output:

----- start =============== XUnitLogChecker Output =====================================================
No dumps found in /home/helixbot/dotnetbuild/dumps.
----- end =============== XUnitLogChecker Output - exit code 0 ===========================

Queue: osx-x64 Debug CoreCLR_Debug
Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=565516&view=logs&j=436e3ba5-8535-542b-4a8e-476703290d56&t=bc25e21f-25dd-5705-c298-9cfd68cf6e1f
Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-66899640128d4698a1/System.Runtime.Numerics.Tests/1/console.139ce6bb.log?helixlogtype=result
Output:

----- start =============== XUnitLogChecker Output =====================================================
No dumps found in /cores.
----- end =============== XUnitLogChecker Output - exit code 0 ===========================

ericstj · 2024-02-14T18:29:44Z

is being found now in outerloop, but it finds no dumps.

I don't see createdump mentioned in the log during the crash like it was before which makes me think the runtime isn't emitting the dmp.

I do see system dumps associated with this failure: https://dev.azure.com/dnceng-public/public/_build/results?buildId=565516&view=ms.vss-test-web.build-test-results-tab&runId=13514696&resultId=108180&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

@hoyosjs I think we need some help understanding why runtime isn't generating the dump specified.

carlossanlop · 2024-02-16T19:55:19Z

I remember XUnitLogChecker is hardcoded to only try to find *.dmp files (unless that has changed recently).

The generated dump file is named in the correct folder: /home/helixbot/dotnetbuild/dumps

But the name is weird: core.1000.22

hoyosjs · 2024-02-16T19:56:02Z

But the name is weird: core.1000.22

that's a system dump

carlossanlop · 2024-02-16T19:56:53Z

Ah okay.

The code is still only looking for *.dmp files:

runtime/src/tests/Common/XUnitLogChecker/XUnitLogChecker.cs

Lines 491 to 499 in cd85023

    
               dumpsFound = Directory 
        
                            .GetFiles(s_configuration.DumpsPath, "*.dmp") 
        
                            .Where(dmp => 
        
                                       DateTime.Compare(File.GetCreationTime(dmp), 
        
                                                        testRunDateTime) >= 0); 
        
           } 
        
           else 
        
           { 
        
               dumpsFound = Directory.GetFiles(s_configuration.DumpsPath, "*.dmp");

…jects to build, check for TargetFrameworkIdentifier==.NETCoreApp instead of BuildTargetFramework==NetCoreAppCurrent.

hoyosjs · 2024-02-24T00:28:04Z

this one looks correct now :)

carlossanlop · 2024-02-26T17:49:02Z

this one looks correct now :)

Hold on, need to run outerloop first. Forgot to submit the command for my latest commits.

carlossanlop · 2024-02-26T17:49:12Z

/azp run runtime-libraries-coreclr outerloop

carlossanlop · 2024-02-26T23:14:17Z

It's working now! I will revert the crash commit and will merge. Here are the logs of the outerloop runs showing that the XUnitLogChecker payload is being included and properly detected, and when possible, it finds the dumps and printing their contents:

OK but affected by XUnitLogChecker: Ensure missing stack frames are resolved on Windows using dotnet-sos #98397:
- windows-x86 Release CoreCLR_Release https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-fef125aecbf348e782/System.IO.Compression.Brotli.Tests/1/console.082205b8.log?helixlogtype=result
- windows-x64 Debug CoreCLR_Debug https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-4afa8b9160034d26a9/System.IO.Compression.Brotli.Tests/1/console.f3dd5101.log?helixlogtype=result
- linux_musl-x64 Debug CoreCLR_Debug https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-8211b8f044044068aa/System.IO.Compression.Brotli.Tests/1/console.d2f23eaa.log?helixlogtype=result
OK but affected by An error occurred trying to start process 'llvm-symbolizer' #91975:
- linux-x64 Debug CoreCLR_Debug https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-dc10b72d530340819f/System.IO.Compression.Brotli.Tests/1/console.04ead2c7.log?helixlogtype=result
OK but affected by XUnitLogChecker: In OSX, no dumps are being found in the /cores directory #94720:
- osx-x64 Debug CoreCLR_Debug https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-1570713915084b3988/System.IO.Compression.Brotli.Tests/1/console.1ec128f5.log?helixlogtype=result

This reverts commit 009b309.

carlossanlop · 2024-02-26T23:16:22Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-02-26T23:16:31Z

Azure Pipelines successfully started running 1 pipeline(s).

carlossanlop · 2024-02-27T01:05:17Z

Looks "good". No issues coming from XUnitLogChecker anymore.

Fixing this is now uncovering a lot of unrelated failures without a KnownBuildIssue. I'll log one for all of them, since this XUnitLogChecker issue was hiding them.

carlossanlop added the area-Infrastructure-libraries label Jan 18, 2024

carlossanlop requested review from ViktorHofer, agocke, akoeplinger, ericstj and hoyosjs January 18, 2024 04:21

carlossanlop self-assigned this Jan 18, 2024

This comment was marked as outdated.

Sign in to view

build-analysis bot mentioned this pull request Jan 18, 2024

XUnitLogChecker: runtime-libraries-coreclr outerloop tests are unable to find the tool DLL #96035

Closed

carlossanlop marked this pull request as ready for review February 14, 2024 01:39

carlossanlop changed the title ~~[Draft] Ensuring XUnitLogChecker is included in libraries outerloop~~ Ensuring XUnitLogChecker is included in libraries outerloop Feb 14, 2024

This was referenced Feb 14, 2024

STATUS_UNSUCCESSFUL in RsaCryptRoundtrip_OaepSHA1 #29683

Open

Tests crashing in CI with no dump: exit code 137 means SIGKILL Killed #97049

Closed

[browser][MT] Assert failed: Cannot find Promise for JSHandle -2 #98406

Closed

This was referenced Feb 16, 2024

[8.0] Failing to build native components #94823

Closed

mono_os_mutex_destroy: pthread_mutex_destroy failed #97565

Closed

Unhandled exception. System.Exception: Profilee returned exit code 139 instead of expected exit code 100 #98500

Closed

ericstj assigned hoyosjs and unassigned carlossanlop Feb 22, 2024

carlossanlop added 2 commits February 23, 2024 13:10

DO NOT MERGE: Testing a forced crash

009b309

To decide whether to include the XUnitLogChecker.csproj among the pro…

0c2fd59

…jects to build, check for TargetFrameworkIdentifier==.NETCoreApp instead of BuildTargetFramework==NetCoreAppCurrent.

hoyosjs approved these changes Feb 24, 2024

View reviewed changes

carlossanlop mentioned this pull request Feb 26, 2024

An error occurred trying to start process 'llvm-symbolizer' #91975

Open

This comment was marked as outdated.

Sign in to view

Revert "DO NOT MERGE: Testing a forced crash"

eeb2ea5

This reverts commit 009b309.

carlossanlop merged commit ac94075 into dotnet:main Feb 27, 2024

carlossanlop deleted the OuterloopXULC branch February 27, 2024 01:29

This was referenced Feb 27, 2024

System.IO.Tests.File_GetSetTimes_SafeFileHandle.WritingShouldUpdateWriteTime_After_SetLastAccessTime failing #97020

Open

profiler\\multiple\\multiple\\multiple.cmd failing on windows arm64 #98817

Closed

github-actions bot locked and limited conversation to collaborators Mar 28, 2024

Ensuring XUnitLogChecker is included in libraries outerloop #97135

Ensuring XUnitLogChecker is included in libraries outerloop #97135

Uh oh!

Conversation

carlossanlop commented Jan 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Jan 18, 2024

Uh oh!

carlossanlop commented Jan 18, 2024

Uh oh!

azure-pipelines bot commented Jan 18, 2024

Uh oh!

carlossanlop commented Jan 18, 2024

Uh oh!

This comment was marked as outdated.

carlossanlop commented Jan 18, 2024

Uh oh!

azure-pipelines bot commented Jan 18, 2024

Uh oh!

carlossanlop commented Feb 14, 2024

Uh oh!

azure-pipelines bot commented Feb 14, 2024

Uh oh!

carlossanlop commented Feb 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericstj commented Feb 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlossanlop commented Feb 16, 2024

Uh oh!

hoyosjs commented Feb 16, 2024

Uh oh!

carlossanlop commented Feb 16, 2024

Uh oh!

hoyosjs commented Feb 24, 2024

Uh oh!

carlossanlop commented Feb 26, 2024

Uh oh!

carlossanlop commented Feb 26, 2024

Uh oh!

This comment was marked as outdated.

carlossanlop commented Feb 26, 2024

Uh oh!

carlossanlop commented Feb 26, 2024

Uh oh!

azure-pipelines bot commented Feb 26, 2024

Uh oh!

carlossanlop commented Feb 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

carlossanlop commented Jan 18, 2024 •

edited

Loading

carlossanlop commented Feb 14, 2024 •

edited

Loading

ericstj commented Feb 14, 2024 •

edited

Loading