Make noop restore faster - speed up evaluations by calling MSBuild Static Graph apis

[jeffkl]

Details about Problem

In MSBuild 16.0, a new Static Graph API was added for parsing project trees in a more efficient way. This new API also can express dependencies so that callers can build a graph that represents the tree. NuGet should adopt this new API to replace the way projects are being evaluated via target execution which would reduce command-line restores dramatically.

NuGet product used (NuGet.exe | VS UI | Package Manager Console | dotnet.exe): Any

NuGet version (x.x.x.xxx): Any

dotnet.exe --version (if appropriate): Any

VS version (if appropriate): Any

OS version (i.e. win10 v1607 (14393.321)): Any

Worked before? If so, with which NuGet version:

Detailed repro steps so we can see the same problem

1. From the command-line, restore a large tree of projects.

The restore takes around 30 seconds for 700 projects. With Static Graph and other perf improvements, it takes less than 6 seconds.

Other suggested things

Verbose Logs

Please include verbose logs (NuGet.exe -verbosity detailed | dotnet.exe --verbosity diag | etc...)

Sample Project

Very helpful if you can zip a project and paste into this issue!

[japj]

Just a check, I had the impression that although the MSBuild static graph api is public/stable now, the implementation is still ongoing (looking at the MSBuild static graph related issue list).

Is the MSBuild implementation now sufficiently stable that static graph is working on all types of solutions and projects? (Or has this the potential to accidentally break nuget restore operations for certain solution/projects?)
perhaps I am missing some insight here? (Although I am very happy at the speed increase)

[jeffkl]

@cdmihai can you comment to the above message?

[cdmihai]

So far static graph is used to successfully build (i.e. call the msbuild build target on) Orchard Core, Orchard classic, Roslyn's Compilers.sln, and the whole Cloudbuild repo (~700 projects, the largest repo we've tried it on). The graph is built following the ProjectReference items. To my knowledge that's what Nuget is also using to build its own graph, so in that respect whatever project graph Nuget produces, the static graph should produce as well.

That being said, these are some caveats that spring in mind:

• graph structure is only based on evaluation time data, not build time data. If there's targets that massage the ProjectReference item at build time, static graph won't see that logic.
• solution support has some missing cases: [Static graph] Instantiate graph from solution dotnet/msbuild#4463 (comment)
• static graph has some special knowledge for multitargeting details and global property inference, which could deviate from reality
• we've never tried it on C++ projects, but AFAIK, nuget doesn't cover C++

[japj]

I know there is a command line MSBuild switch for using static graph/isolation projects.
But shouldn’t there be a VisualStudio setting to turn this behavior on/off for trying out with projects?

Related to nuget support, that is implemented through some Microsoft common targets mechanism and nuget restore is “technically” available in c++ projects too (e.g. nuget MSBuild tools packages can be consumed in c++ projects, although the VS UI doesn’t handle it)

FYI we have mixed project type solutions (c# old/new style, sqldb, c++, powershell, Azure, SSIS, etc) incl. some customizations like using the Microsoft CentralPackageVersioning MSBuild Sdk.

[cdmihai]

I know there is a command line MSBuild switch for using static graph/isolation projects.
But shouldn’t there be a VisualStudio setting to turn this behavior on/off for trying out with projects?

It depends if VS wants to explore this or not. /graph might make a solution build faster (when VS invokes msbuild on the solution and lets msbuild manage the graph). /isolate is more useful to ensure correctness (ensure the static graph does not have gaps, ensure that higher order build engines (BuildXL and Quickbuild) behave exactly like vanilla msbuild, etc) and to ensure that a project is built in isolation of other projects. VS could also replace it's own msbuild graph with static graph to reduce code duplication.

Related to nuget support, that is implemented through some Microsoft common targets mechanism and nuget restore is “technically” available in c++ projects too (e.g. nuget MSBuild tools packages can be consumed in c++ projects, although the VS UI doesn’t handle it)

FYI we have mixed project type solutions (c# old/new style, sqldb, c++, powershell, Azure, SSIS, etc) incl. some customizations like using the Microsoft CentralPackageVersioning MSBuild Sdk.

As long as projects use ProjectReference (with the standard metadata) to express dependencies, constructing a static graph should be fine, indifferent or project system, sdk, or homogeneity. Building the graph with vanilla msbuild correctness parity is another issue, but for nuget restore that feature is not used.