Proposal: Robust dependency management #261
Comments
hsyed-dojo
changed the title
|
I've done some thinking on this topic over in @rules_msbuild, thought I'd share what I've learned: Bazel brings a unique challenge to the table for using Directory.Packages.props: Take NewtonSoft.Json for example: .NETStandard 1.0 depends on:
Directory.Packages.props only specifies the PackageId, but does not specify the frameworks to restore for. If a workspace only uses Netstandard2.0, and an implementation fetches the packages ahead of time without knowing what projects it is fetching for, then a naive implementation could end up fetching a bunch of NuGet dependencies for Netstandard1.0 that arent needed. Over in @rules_msbuild, I accommodated this with a similar approach to Directory.Packages.props, but with a slight tweak: I specify the PackageId, and then the list of frameworks that depend on it. This way, packages can be fetched ahead of time without loading build files, and no unnecessary packages are fetched. Example format automatically managed by @rules_msbuild//gazelle/dotnet nuget_fetch(
name = "nuget",
packages = {
"NewtonSoft.Json/13.0.1": ["net5.0", "netstandard2.1"], # no transitive dependencies will be fetched
}
)Given that a Directory.Packages.props file is just another project file to msbuild, this likely could also be handled by simply added metadata to the Package item, so long as that metadata name doesn't conflict with MSBuild's normal metadata for Packages: <Project>
<ItemGroup>
<PackageReference Include="NewtonSoft.Json" Version="13.0.1">
<TargetFrameworks>net5.0,netstandard2.1</TargetFrameworks>
</PackageReference>
</ItemGroup>
</Project>It would be a slight departure from the standard feature set of the Directory.Packages.props file, but definitely something that an automated tool could manage. @rules_msbuild//gazelle/dotnet would certainly be capable of outputting into this format with a little work on merging the XML files. Automation is ideal for this type of Package management that specifies the TargetFramework, because the user would be required to maintain the TargetFramework in two places: once in the project file of the target assembly, and another place for bazel to know what dependency graph to fetch ahead of time. Alternatively, the nuget tool could somehow parse all of the project files in the workspace that have packages specified, but that seems to be counter to Bazels approach of Loading, Analysis, and Execution separations. You would also most likely have to actually evaluate the project files, which is starting to encroach on performing an actual build. Just my two cents! |
|
Hi @samhowes, thanks for your input! We have had the luck to not require multi-targeting (frameworks & runtimes) until now. In our approach we just managed THE TargetFramework from the nuget repository rule Currently I have a draft on Source Generators however which require to be built against netstandard2.0. My current approach here is just allowing a list of I'm not sure if this is actually a problem when we implement the pinning behavior of rules_jvm_external. This would mean persisting the restore graph in a lock/pin file and only downloading the necessary nugets via I think a combination of default behavior, that uses the standard Props file, with the ability to narrow it using |
|
I would like to come with a twist here! In the F# community there is a pretty popular package manager that has all these features out of the box (lock file, multi targeting, git references, etc.): https://fsprojects.github.io/Paket/ I'm using a small CLI tool that I wrote to consumer the Paket lockfile as an alternative to the current If anybody is interested in this approach I can share the code with not too much effort. |
|
E.g. if I have this I get this |
|
@purkhusid Interesting to hear that youv'e found packet easier to work with compared to others tools, I haven't checked it out yet, but I think I will now! Just want to clarify one thing: I.e. Foo.csproj is a ConsoleApp that targets net5.0, but Bar.csproj is a nuget library that targets Netstandard1.0 If Bar.csproj does not depend on Newtonsoft.Json, then Newtonsoft.Json should not be restored for Netstandard1.0. But if Bar.csproj depends on Serilog then we need to download the deps for serilog. Does paket support that? @tomdegoede That sounds like some convenient constraints you have! At my day-job, we have some projects that target netcoreapp2.2, some projects that target net5.0, some projects that target netstandard2.0, and other projects that target netstandard2.1, and most target netcoreapp3.1. The projects that target netcoreapp2.2 are still on EntityFramework/2.2.6, and the other projects are on EntityFramework/3.1.4. So having a solution that supports multiple nuget package versions and restored for different frameworks is important for me. The projects with different EF versions are in separate "sections" of the codebase, so we don't have to worry about a netcoreapp2.2 project getting EF3.1 i.e. diamond dependencies. My focus in @rules_msbuild has been making a "drop-in replacement for dotnet build", that may not be the focus of this proposal though. With a packages.lock file, don't we have a chicken-and-egg problem? If the user has never restored packages, how do they get the packages.lock file to begin with? Sure, they could do I like the idea of the lock file restore for a more efficient restore and guaranteed restore graph though. |
|
@samhowes thank you for bringing this up, I'm learning the finer details as I go along. An example rule generated by the nuget_package(
name = "grpc.core",
package = "grpc.core",
version = "2.28.1",
sha256 = "b625817b7e8dfe66e0894b232001b4c2f0e80aa41dc4dccb59d5a452ca36a755",
core_lib = {
"netcoreapp2.0": "lib/netstandard2.0/Grpc.Core.dll",
"netcoreapp2.1": "lib/netstandard2.0/Grpc.Core.dll",
"netcoreapp2.2": "lib/netstandard2.0/Grpc.Core.dll",
"netcoreapp3.0": "lib/netstandard2.0/Grpc.Core.dll",
"netcoreapp3.1": "lib/netstandard2.0/Grpc.Core.dll",
"net5.0": "lib/netstandard2.0/Grpc.Core.dll",
},
core_deps = {
"netcoreapp2.0": [
"@grpc.core.api//:netcoreapp2.0_core",
"@system.memory//:netcoreapp2.0_core",
],
"netcoreapp2.1": [
"@grpc.core.api//:netcoreapp2.1_core",
"@system.memory//:netcoreapp2.1_core",
],
"netcoreapp2.2": [
"@grpc.core.api//:netcoreapp2.2_core",
"@system.memory//:netcoreapp2.2_core",
],
"netcoreapp3.0": [
"@grpc.core.api//:netcoreapp3.0_core",
"@system.memory//:netcoreapp3.0_core",
],
"netcoreapp3.1": [
"@grpc.core.api//:netcoreapp3.1_core",
"@system.memory//:netcoreapp3.1_core",
],
"net5.0": [
"@grpc.core.api//:net5.0_core",
"@system.memory//:net5.0_core",
],
},
core_files = {
"netcoreapp2.0": [
"lib/netstandard2.0/Grpc.Core.dll",
"lib/netstandard2.0/Grpc.Core.pdb",
"lib/netstandard2.0/Grpc.Core.xml",
"runtimes/linux/native/libgrpc_csharp_ext.x64.so",
"runtimes/linux/native/libgrpc_csharp_ext.x86.so",
"runtimes/osx/native/libgrpc_csharp_ext.x64.dylib",
"runtimes/osx/native/libgrpc_csharp_ext.x86.dylib",
],
"netcoreapp2.1": [
"lib/netstandard2.0/Grpc.Core.dll",
"lib/netstandard2.0/Grpc.Core.pdb",
"lib/netstandard2.0/Grpc.Core.xml",
"runtimes/linux/native/libgrpc_csharp_ext.x64.so",
"runtimes/linux/native/libgrpc_csharp_ext.x86.so",
"runtimes/osx/native/libgrpc_csharp_ext.x64.dylib",
"runtimes/osx/native/libgrpc_csharp_ext.x86.dylib",
],
"netcoreapp2.2": [
"lib/netstandard2.0/Grpc.Core.dll",
"lib/netstandard2.0/Grpc.Core.pdb",
"lib/netstandard2.0/Grpc.Core.xml",
"runtimes/linux/native/libgrpc_csharp_ext.x64.so",
"runtimes/linux/native/libgrpc_csharp_ext.x86.so",
"runtimes/osx/native/libgrpc_csharp_ext.x64.dylib",
"runtimes/osx/native/libgrpc_csharp_ext.x86.dylib",
],
"netcoreapp3.0": [
"lib/netstandard2.0/Grpc.Core.dll",
"lib/netstandard2.0/Grpc.Core.pdb",
"lib/netstandard2.0/Grpc.Core.xml",
"runtimes/linux/native/libgrpc_csharp_ext.x64.so",
"runtimes/linux/native/libgrpc_csharp_ext.x86.so",
"runtimes/osx/native/libgrpc_csharp_ext.x64.dylib",
"runtimes/osx/native/libgrpc_csharp_ext.x86.dylib",
],
"netcoreapp3.1": [
"lib/netstandard2.0/Grpc.Core.dll",
"lib/netstandard2.0/Grpc.Core.pdb",
"lib/netstandard2.0/Grpc.Core.xml",
"runtimes/linux/native/libgrpc_csharp_ext.x64.so",
"runtimes/linux/native/libgrpc_csharp_ext.x86.so",
"runtimes/osx/native/libgrpc_csharp_ext.x64.dylib",
"runtimes/osx/native/libgrpc_csharp_ext.x86.dylib",
],
"net5.0": [
"lib/netstandard2.0/Grpc.Core.dll",
"lib/netstandard2.0/Grpc.Core.pdb",
"lib/netstandard2.0/Grpc.Core.xml",
"runtimes/linux/native/libgrpc_csharp_ext.x64.so",
"runtimes/linux/native/libgrpc_csharp_ext.x86.so",
"runtimes/osx/native/libgrpc_csharp_ext.x64.dylib",
"runtimes/osx/native/libgrpc_csharp_ext.x86.dylib",
],
},
)Framework specific elements seem to be modelled comprehensively. I Agree with @tomdegoede, the cost of resolving extra frameworks might be ok in the short term IMO and we can make it more granular at a later stage if needed. Developers maintaining the build already have everything resolved so the UX should be fine. Finer granularity upfront means manual maintenance overhead or the development of tooling now. I think we should have a top level list of frameworks for which to resolve the transitive deps. We could extend the dotnet_resolve_dependencies(
targetFrameworks = ["netcore5.0", "netcore2.1"],
dependencies_file = "Directories.Packages.props",
lock_file = "directories_packages_lock.json"
)
dotnet_install_depdendencies(
name = "nuget",
lock_file = "directories_packages_lock.json"
) |
|
@samhowes thats my bad. I misused the term multi-targeting for the more general "having to worry about multiple target platforms / runtimes". Your scenario is very interesting. I think that having multiple versions of the same dependency is sort of a Bazel anti-pattern. Nonetheless we should be able to support this using multiple Directory.Build.Props resulting in differente We could also argue that we can reduce the complexity of a single nuget_repository to a single targetFramework. When we target multiple frameworks we can then just configure multiple nuget_repositories like |
|
@samhowes It is supported by Paket to have multiple dependency groups: https://fsprojects.github.io/Paket/groups.html In my CLI I generate different targets for each group: E.g. if this were the paket.dependencies file: Then my tool generates packages for each group that are accessible as: |
|
@purkhusid Would you be able to open-source this tool? I am interested in trying it. |
|
@njlr Do you specifically want to use Paket or do you just need a better tool for NuGet packages in Bazel? The tool is not currently in shape for sharing it but I might have time soon to clean it up and share it. We are looking into using a fairly new feature in NuGet which allows centrally managing packages and generating a lock file. If we'll be successful we'll probably make that the default way for 3rd party dependencies in rules_dotnet |
|
@njlr Ok, I'll take a look at this as soon as I can. My bandwidth is fairly limited at the moment but I hope I can do some rules_dotnet work soon. |
|
The Paket support works very well at the moment and is now on |
|
I've been looking at implementing a solution based on I was able to write a small script that converts the lock file into a Just wanted to post this in case someone's interested in this approach as an alternative to paket. |
|
I'm going to close this issue since the Paket approach is well supported at the moment. Feel free to open a new issue specifically for a non-paket solution. |
Robust dependency management
Participants: @hsyed-dojo and @tomdegoede
Reviewers: @purkhusid
Dependency management is arguably the most complex aspect of managing a Bazel build at scale. A dependency management solution is needed that will work ergonomically in a dual build setup (msbuild and bazel). This is one of
the ingredients for full build generation #258 and is one of the changes we would like to bring in for increased adoption
of these official rules #260.
We propose a solution that follows the developer experience, mechanics and patterns in use by
rules_jvm_external.
Dotnet builds have a convention in the works for centralising the dependencies of a project. A top level
Directory.Packages.propsis a manifest for all the coordinates and versions in use by asolution. We propose that this file become the standard way of encoding dependencies for projects
that use these rules.
Core points:
@nuget.builds should always be pinned.
rules_jvm_externalwe will have a custom/expanded lock file format to populatethe dependencies. An expanded format would be preferable because we can then share this with msbuild.
The text was updated successfully, but these errors were encountered: