Local live-live builds (dotnet#494)

* Convert libraries and installer to use a live CoreCLR.

* Installer uses a live libraries build for DLLs.

* Remove newline in path.

* Enable live-live builds to build required live dependencies when the dependencies are in the building subset/subset category.

* Update path to coreclr output.

* Fix typo in coreclr.proj.

* Remove last usage of CoreCLRPDBOverridePath.

* Update path of binplaced runtime to include OS and Arch.

* Fix typo in coreclr/tests/src/Common/Directory.Build.targets.

* Restore CoreFX shared framework from live build for CoreCLR test dependencies.

* Remove dependency on Microsoft.NETCore.CoreCLR.TestDependencies package.

* Enable referencing live builds of the OOB libraries by temporarily adding a new BinplaceConfiguration. Update test_dependencies in the CoreCLR tree to use that to restore OOB libraries.

* Remove reference to System.Drawing.Common from NETClientPrimitives (the reference is only needed at runtime and Core_Root already has System.Drawing.Common).

* Remove package-based checked CoreCLR + CoreFX test runs. In the live-live world, we'll run this test configuration by pointing the CoreFX test host build step at the artifacts from a checked CoreCLR build (possible in both local and CI builds).

* Only copy the UCRT on Windows.

* CoreCLR test build now builds against a live libraries sfx+oob ref assembly set.

* Override GetCorePackagePaths target to not reference unused packages (breaks the live-live build on a clean system).

* Remove live references to Microsoft.NETCore.Platforms now that it seems to be unneeded since the SDK now packages a RID graph. Use the live RID graph in test_runtime.csproj where we may actually be looking for the live RID graph.

* Fix coreclr build arg generation bug that was causing build failures.

* Remove references to Microsoft.NETCore.Platforms outside of package references from the installers.

* Fix problems in installer build in live-live that were found from a clean build.

* Remove unused dependency on Microsoft.NETCore.Targets.

* Fix Microsoft.NETCore.App -> Microsoft.NETCore.Platforms dependency.

* Remove Microsoft.NETCore.Targets version from Versions.props.

* Remove unused deps.csproj file that was the last dependency on the packaged Microsoft.NETCore.Platforms package.

* Fix OSGroup calculation for dependent projects off-Windows.

* Fix construction of Microsoft.NETCore.App.Internal package.

* Remove unneeded CoreRun copying.

* Serialize subsets that have dependencies across subset groups. Don't use `<MSBuild />` tasks to rebuild subsets since we don't want to rebuild projects multiple times (coreclr was being built multiple times).

* Cleanup from PR feedback.

* Only copy libraries outputs to artifacts/bin/runtime.

* Use the netcoreapp output directories for OOB-included references to libraries outputs.

* Remove references to deleted old corefx override setup.

* First pass to get new runtime pipeline using the local live-live infra (coreclr+libraries only for now).

* Don't generate layout during test native build.

* Fix ref assembly default binplacing for netcoreapp.

* Remove msbuild goo around importing Subsets.props and Directory.Build.props only once each. This goo isn't needed anymore.

* Fix disabling of binplacing for the target vertical to exclude only runtime.depproj and not exclude winrt.depproj.

* Fix libraries test build.

* Copy all paths used by the "ResolveLibrariesFromLocalBuild" target to publish from libraries build.

* For shims, reference the System.Private.CoreLib directly from CoreCLR outputs instead of through runtime.depproj.

* Hard-code the libraries configuration to release in coreclr test builds.

* Resolve ReferenceFromRuntime items directly from the CoreCLR output instead of via the runtime.depproj project.

* Fix test host generation. Fix running the XML Serialization generator as part of libraries test build.

* Fix XmlSerializer.Generator.Tests again.

* Don't look for CoreCLR files when using ReferenceFromRuntime and not using netcoreapp.

* Crossgen-comparison should depend on a libraries build so we can populate Core_Root.

* Add job dependency on libraries build from run-test-job to ensure we wait correctly. Otherwise we might have a race since the managed test build might depend on one libraries build while the test run depends on another.

* Only resolve coreclr files from local build when targetting netcoreapp.

* Add steps to download libraries build for crossgen-comparison.

* Revive old method of resolving references to the runtime for building against old netcoreapp (ie netcoreapp 3.0). Fix correctly passing along aliases info.

* Fix failing AllConfigurations leg that was due to the netcoreapp runtime.depproj getting binplaced in the runtime output folder.

* Update eng/liveBuilds.targets

Co-Authored-By: Davis Goodin <[email protected]>

* HostArch->BuildArchitecture and share with installer.

* Fix format nits.

* Update YAML variable name.

* Remove setting property to import liveBuilds.targets.

* Upload docs and packages folders from libraries build

* Inclue doc files in the Microsoft.NETCore.App.Internal package

* Try making the runtime-installer pipeline live-live using the build-coreclr-and-libraries-job template.

* Update artifacts download paths and msbuild properties for installer build.

* Mark cross-component files as native.

* Remove FrameworkPackageName to  remove error about missing $(FrameworkPackageName).versions.txt file that other shared frameworks also don't generate.

* Restore GetFilesFromPackageResolve override.

* Don't clean the unpack folder for the libraries download. It wipes out the CoreCLR download.

* Make sure that we restore the WinRT targetting pack during the installer build.

* Add libraries package directory to RestoreSources for the case where the LibrariesConfiguration is different from the installer build configuration.

* Don't pass OfficialBuildId unless it is an official build. We want to be consistent with the libraries build so our package versions match.

* Give default value for OfficialBuildArg variable

* Ensure installer tests can see the libraries-generated packages. Fix typo in TargetPath for CoreCLR assets (runtime instead of runtimes).

Fix paths for CoreCLR cross-target files as well as crossgen to place them in the correct directories.

* Add back FrameworkPackageName to get the partial platform manifest to generate.

Author: jkoritzinsky

Directory.Build.props

@@ -60,6 +60,7 @@
     <RepositoryUrl>git://github.com/dotnet/runtime</RepositoryUrl>
     <ProjectUrl>https://github.com/dotnet/runtime</ProjectUrl>
     <LicenseUrl>https://github.com/dotnet/runtime/blob/master/LICENSE.TXT</LicenseUrl>
+    <RuntimeIdGraphDefinitionFile>$([MSBuild]::NormalizePath('$(LibrariesProjectRoot)', 'pkg', 'Microsoft.NETCore.Platforms', 'runtime.json'))</RuntimeIdGraphDefinitionFile>
     <LicenseFile>$(RepoRoot)LICENSE.TXT</LicenseFile>
 
     <!-- Indicates this is not an officially supported release. Release branches should set this to false. -->
@@ -89,6 +90,8 @@
     <PortableBuild Condition="'$(PortableBuild)' == ''">true</PortableBuild>
     <!-- Used for launchSettings.json and runtime config files. -->
     <AppDesignerFolder>Properties</AppDesignerFolder>
+    <!-- Determine what architecture we are building on. -->
+    <BuildArchitecture>$([System.Runtime.InteropServices.RuntimeInformation]::ProcessArchitecture.ToString().ToLowerInvariant())</BuildArchitecture>
   </PropertyGroup>
 
   <Import Project="$(RepositoryEngineeringDir)Analyzers.props" />

Directory.Build.targets

@@ -8,6 +8,7 @@
     <ImportDirectoryBuildTargets>false</ImportDirectoryBuildTargets>
   </PropertyGroup>
 
+  <Import Project="$(RepositoryEngineeringDir)/liveBuilds.targets" />
   <Import Project="Sdk.targets" Sdk="Microsoft.DotNet.Arcade.Sdk" Condition="'$(SkipImportArcadeSdkFromRoot)' != 'true'" />
 
   <PropertyGroup>
@@ -18,4 +19,4 @@
     <NETCoreAppMaximumVersion>$(MajorVersion).$(MinorVersion)</NETCoreAppMaximumVersion>
   </PropertyGroup>
 
-</Project>
+</Project>

eng/Subsets.props

@@ -76,7 +76,7 @@
 
     <!-- Libraries -->
     <SubsetName Include="All" Category="Libraries" Description="The .NET libraries comprising the shared framework." />
-    
+
     <!-- CoreClr -->
     <SubsetName Include="All" Category="CoreClr" Description="The .NET runtime." />
 
@@ -96,14 +96,16 @@
     <ProjectToBuild Include="$(RepoToolsLocalDir)regenerate-readme-table.proj" />
   </ItemGroup>
 
-  <!-- Libraries sets -->
-  <ItemGroup Condition="$(_subsetCategory.Contains('libraries')) and $(_subset.Contains('all'))">
-    <ProjectToBuild Include="$(LibrariesProjectRoot)build.proj" />
-  </ItemGroup>
-
   <!-- CoreClr sets -->
   <ItemGroup Condition="$(_subsetCategory.Contains('coreclr')) and $(_subset.Contains('all'))">
-    <ProjectToBuild Include="$(CoreClrProjectRoot)coreclr.proj" />
+    <CoreClrProjectToBuild Include="$(CoreClrProjectRoot)coreclr.proj" BuildInParallel="false" />
+    <ProjectToBuild Include="@(CoreClrProjectToBuild)" />
+  </ItemGroup>
+
+  <!-- Libraries sets -->
+  <ItemGroup Condition="$(_subsetCategory.Contains('libraries')) and $(_subset.Contains('all'))">
+    <LibrariesProjectToBuild Include="$(LibrariesProjectRoot)build.proj" BuildInParallel="false" />
+    <ProjectToBuild Include="@(LibrariesProjectToBuild)" />
   </ItemGroup>
 
   <!-- Installer sets -->
@@ -119,23 +121,23 @@
   </ItemGroup>
 
   <ItemGroup Condition="$(_subsetCategory.Contains('installer')) and $(_subset.Contains('depproj'))">
-    <DepprojProjectToBuild Include="$(InstallerProjectRoot)pkg\projects\**\*.depproj" SignPhase="Binaries" />
+    <DepprojProjectToBuild Include="$(InstallerProjectRoot)pkg\projects\**\*.depproj" SignPhase="Binaries" BuildInParallel="false" />
     <ProjectToBuild Include="@(DepprojProjectToBuild)" />
   </ItemGroup>
 
   <ItemGroup Condition="$(_subsetCategory.Contains('installer')) and $(_subset.Contains('pkgproj'))">
-    <PkgprojProjectToBuild Include="$(InstallerProjectRoot)pkg\projects\**\*.pkgproj" SignPhase="MsiFiles" />
+    <PkgprojProjectToBuild Include="$(InstallerProjectRoot)pkg\projects\**\*.pkgproj" SignPhase="MsiFiles" BuildInParallel="false" />
     <ProjectToBuild Include="@(PkgprojProjectToBuild)" />
   </ItemGroup>
 
   <ItemGroup Condition="$(_subsetCategory.Contains('installer')) and $(_subset.Contains('bundle'))">
-    <BundleProjectToBuild Include="$(InstallerProjectRoot)pkg\projects\**\*.bundleproj" SignPhase="BundleInstallerFiles" />
+    <BundleProjectToBuild Include="$(InstallerProjectRoot)pkg\projects\**\*.bundleproj" SignPhase="BundleInstallerFiles" BuildInParallel="false" />
     <ProjectToBuild Include="@(BundleProjectToBuild)" />
   </ItemGroup>
 
   <ItemGroup Condition="$(_subsetCategory.Contains('installer')) and $(_subset.Contains('installers'))">
-    <InstallerProjectToBuild Include="$(InstallerProjectRoot)pkg\packaging\installers.proj" />
-    <InstallerProjectToBuild Include="$(InstallerProjectRoot)pkg\packaging\vs-insertion-packages.proj" />
+    <InstallerProjectToBuild Include="$(InstallerProjectRoot)pkg\packaging\installers.proj" BuildInParallel="false" />
+    <InstallerProjectToBuild Include="$(InstallerProjectRoot)pkg\packaging\vs-insertion-packages.proj" BuildInParallel="false" />
     <ProjectToBuild Include="@(InstallerProjectToBuild)" />
   </ItemGroup>
 
@@ -147,7 +149,7 @@
     <TestProjectToBuild Include="$(InstallerProjectRoot)test\HostActivation.Tests\HostActivation.Tests.csproj" />
     <TestProjectToBuild Include="$(InstallerProjectRoot)test\Microsoft.DotNet.CoreSetup.Packaging.Tests\Microsoft.DotNet.CoreSetup.Packaging.Tests.csproj" />
     <TestProjectToBuild Include="$(InstallerProjectRoot)test\Microsoft.Extensions.DependencyModel.Tests\Microsoft.Extensions.DependencyModel.Tests.csproj" />
-    <ProjectToBuild Include="@(TestProjectToBuild)" />
+    <ProjectToBuild Include="@(TestProjectToBuild)" BuildInParallel="false" />
   </ItemGroup>
 
 </Project>

eng/Tools.props

@@ -13,14 +13,11 @@
     <!-- arcade -->
     <PackageReference Include="Microsoft.DotNet.GenFacades" Version="$(MicrosoftDotNetGenFacadesVersion)" />
     <PackageReference Include="Microsoft.DotNet.Build.Tasks.Packaging" Version="$(MicrosoftDotNetBuildTasksPackagingVersion)" />
-    
+
     <!-- coreclr -->
     <!-- Download the package in the initial arcade restore step to work around race conditions when restoring an msbuild SDK. -->
     <PackageDownload Include="Microsoft.NET.Sdk.IL" Version="[$(MicrosoftNETSdkILVersion)]" />
 
-    <!-- corefx -->
-    <PackageReference Include="Microsoft.NETCore.Platforms" Version="$(MicrosoftNETCorePlatformsVersion)" />
-
     <!-- roslyn -->
     <PackageReference Include="Microsoft.Net.Compilers.Toolset" Version="$(MicrosoftNetCompilersToolsetVersion)" PrivateAssets="all" IsImplicitlyDefined="true" />
 

eng/Version.Details.xml

@@ -1,13 +1,5 @@
 <Dependencies>
   <ProductDependencies>
-    <Dependency Name="Microsoft.NETCore.Platforms" Version="5.0.0-alpha.1.19563.6">
-      <Uri>https://github.com/dotnet/corefx</Uri>
-      <Sha>5cee7c97d602f294e27c582d4dab81ec388f1d7b</Sha>
-    </Dependency>
-    <Dependency Name="Microsoft.NETCore.Targets" Version="5.0.0-alpha.1.19563.6">
-      <Uri>https://github.com/dotnet/corefx</Uri>
-      <Sha>5cee7c97d602f294e27c582d4dab81ec388f1d7b</Sha>
-    </Dependency>
     <Dependency Name="Microsoft.Private.CoreFx.NETCoreApp" Version="5.0.0-alpha.1.19563.6">
       <Uri>https://github.com/dotnet/corefx</Uri>
       <Sha>5cee7c97d602f294e27c582d4dab81ec388f1d7b</Sha>
@@ -28,10 +20,6 @@
       <Uri>https://github.com/dotnet/corefx</Uri>
       <Sha>cf64918877d98577363bb40d5eafac52beb80a79</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.NETCore.Runtime.CoreCLR" Version="5.0.0-alpha1.19563.3">
-      <Uri>https://github.com/dotnet/coreclr</Uri>
-      <Sha>2c4fb3250989f014550882f5d165cdc36ebdbd08</Sha>
-    </Dependency>
     <Dependency Name="Microsoft.NETCore.ILAsm" Version="5.0.0-alpha1.19563.3">
       <Uri>https://github.com/dotnet/coreclr</Uri>
       <Sha>2c4fb3250989f014550882f5d165cdc36ebdbd08</Sha>

eng/Versions.props

@@ -49,12 +49,8 @@
     <MicrosoftDotNetPlatformAbstractionsVersion>3.0.0</MicrosoftDotNetPlatformAbstractionsVersion>
     <!-- CoreClr dependencies -->
     <MicrosoftNETCoreILAsmVersion>5.0.0-alpha1.19563.3</MicrosoftNETCoreILAsmVersion>
-    <MicrosoftNETCoreRuntimeCoreCLRVersion>5.0.0-alpha1.19563.3</MicrosoftNETCoreRuntimeCoreCLRVersion>
     <MicrosoftNETSdkILVersion>5.0.0-alpha1.19563.3</MicrosoftNETSdkILVersion>
     <!-- Libraries dependencies -->
-    <MicrosoftNETCorePlatformsVersion>5.0.0-alpha.1.19563.6</MicrosoftNETCorePlatformsVersion>
-    <MicrosoftPrivateCoreFxNETCoreAppVersion>5.0.0-alpha.1.19562.8</MicrosoftPrivateCoreFxNETCoreAppVersion>
-    <MicrosoftNETCoreTargetsVersion>5.0.0-alpha.1.19563.6</MicrosoftNETCoreTargetsVersion>
     <SystemTextJsonVersion>5.0.0-alpha.1.19563.6</SystemTextJsonVersion>
     <SystemTextEncodingsWebVersion>5.0.0-alpha.1.19563.6</SystemTextEncodingsWebVersion>
     <runtimenativeSystemIOPortsVersion>5.0.0-alpha.1.19563.3</runtimenativeSystemIOPortsVersion>
@@ -106,8 +102,6 @@
     <LinuxCoreFxOptimizationDataPackage>optimization.linux-x64.ibc.corefx</LinuxCoreFxOptimizationDataPackage>
     <MicrosoftPrivateIntellisensePackage>microsoft.private.intellisense</MicrosoftPrivateIntellisensePackage>
     <MicrosoftDotNetXUnitConsoleRunnerPackage>Microsoft.DotNet.XUnitConsoleRunner</MicrosoftDotNetXUnitConsoleRunnerPackage>
-    <MicrosoftPrivateCoreFxNETCoreAppPackage>Microsoft.Private.CoreFx.NETCoreApp</MicrosoftPrivateCoreFxNETCoreAppPackage>
-    <MicrosoftNETCorePlatformsPackage>Microsoft.NETCore.Platforms</MicrosoftNETCorePlatformsPackage>
     <MicrosoftNETCoreAppPackage>Microsoft.NETCore.App</MicrosoftNETCoreAppPackage>
     <optimizationIBCCoreCLRPackage>optimization.IBC.CoreCLR</optimizationIBCCoreCLRPackage>
     <optimizationPGOCoreCLRPackage>optimization.PGO.CoreCLR</optimizationPGOCoreCLRPackage>