Skip to content

Latest commit

 

History

History
378 lines (300 loc) · 17.7 KB

BUILD-INTEGRATION.md

File metadata and controls

378 lines (300 loc) · 17.7 KB
Raw

Protocol Buffers/gRPC Integration Into .NET Build

With Grpc.Tools package version 1.17 we made it easier to compile .proto files in your project using the dotnet build command, Visual Studio, or command-line MSBuild. You need to configure the .csproj project according to the way you want to integrate Protocol Buffer files into your build. If you are upgrading an existing project, read through this list of common scenarios and decide if any one of them matches your approach. The protoc command line migration is explained near the end of this document; this migration may be the quickest but not the long-term solution.

There is also a Reference section at the end of the file.

Reporting issues

First thing first, if you found a bug in this new build system, or have a scenario that is not easily covered, please open an issue in the gRPC repository, and tag the user @kkm000 somewhere in the text (for example, include /cc @kkm000 at end of the issue text) to seize his immediate attention.

Common scenarios

I just want to compile .proto files into my library

This is the approach taken by the examples in the csharp/examples directory. Protoc output files (for example, Helloworld.cs and HelloworldGrpc.cs compiled from helloworld.proto) are placed among object and other temporary files of your project, and automatically provided as inputs to the C# compiler. As with other automatically generated .cs files, they are included in the source and symbols NuGet package, if you build one.

Simply reference your .proto files in a <Protobuf> item group. The following example will add all .proto files in a project and all its subdirectories (excluding special directories such as bin and obj):

  <ItemGroup>
    <Protobuf Include="**/*.proto" />
  </ItemGroup>

You must add a reference to the NuGet packages Grpc.Tools and Grpc (the latter is a meta-package, in turn referencing Grpc.Core and Google.Protobuf packages). It is very important to mark Grpc.Tools as a development-only dependency, so that the users of your library do not fetch the tools package:

  • "Classic" .csproj with packages.config (Visual Studio, Mono): This is handled automatically by NuGet. See the attribute added by Visual Studio to the packages.config file in the HelloworldLegacyCsproj/Greeter example.

  • "SDK" .csproj (Visual Studio, dotnet new): Add an attribute PrivateAssets="All" to the Grpc.Tools package reference. See an example in the Greeter.csproj example project in this repository. If adding a package reference in Visual Studio, edit the project file and add this attribute. This is a bug in NuGet client.

If building a NuGet package from your library with the nuget command line tool from a .nuspec file, then the spec file may (and probably should) reference the Grpc metapackage, but do not add a reference to Grpc.Tools to it. .NET "SDK" projects handle this automatically when called from dotnet pack by excluding any packages with private assets, such as thus marked Grpc.Tools.

Per-file options that can be set in Visual Studio

For a "classic" project, you can only add .proto files with all options set to default (if you find it necessary to modify these options, then hand-edit the .csproj file). Click on the "show all files" button, add files to project, then change file type of the .proto files to "Protobuf" in the Properties window drop-down. This menu item will appear after you import the Grpc.Tools package:

Properties in a classic project

For an "SDK" project, you have more control of some frequently used options. You may need to open and close Visual Studio for this form to appear in the properties window after adding a reference to Grpc.Tools package (we do not know whether this is a bug or by design, but it looks like a bug):

Properties in an SDK project

You can also change options of multiple files at once by selecting them in the Project Explorer together.

See the Reference section at end of this file for options that can be set per-file by modifying the source .csproj directly.

My .proto files are in a directory outside the project

Refer to the example files RouteGuide.csproj and Greeter.csproj in this repository. For the files to show up in Visual Studio properly, add a Link attribute with just a filename to the <Protobuf> item. This will be the display name of the file. In the Include attribute, specify the complete path to file. A relative path is based off the project directory.

Or, if using Visual Studio, add files as links from outside directory. In the Add Files dialog, there is a little down arrow near the Open button. Click on it, and choose "Add as link". If you do not select this option, Visual Studio will copy files to the project directory instead.

I just want to generate proto and gRPC C# sources from my .proto files (no C# compile)

Suppose you want to place generated files right beside each respective source .proto file. Create a .csproj library file in the common root of your .proto tree, and add a reference to Grpc.Tools package (this works in Windows too, $ below stands for a command prompt in either platform):

/myproject/myprotofiles$ dotnet new classlib
  . . .
  Restoring packages for /myproject/myprotofiles/myprotofiles.csproj...
  . . .
/myproject/myprotofiles$ rm *.cs        <-- remove all *.cs files from template;
C:\myproject\myprotofiles> del *.cs /y  <-- on Windows, use the del command instead.
/myproject/myprotofiles$ dotnet add package Grpc.Tools

(the latter command also accepts an optional --version X.Y switch for a specific version of package, should you need one). Next open the generated .csproj file in a text editor.

Since you are not building a package, you may not worry about adding PrivateAssets="All" attribute, but it will not hurt, in case you are repurposing the project at some time later. The important part is (1) tell the gRPC tools to select the whole directory of files; (2) order placement of each output besides its source, and (3) not compile the generated .cs files. Add the following stanza under the <Project> xml node:

  <ItemGroup>
    <Protobuf Include="**/*.proto" OutputDir="%(RelativePath)" CompileOutputs="false"  />
  </ItemGroup>

The Include tells the build system to recursively examine project directory and its subdirectories (**) include all files matching the wildcard *.proto. You can instead selectively include your files or selectively exclude files from the glob pattern; MSBuild documentation explains that. The OutputDir="%(RelativePath)" orders the output directory for each .cs file be same as the corresponding .proto directory. Finally, CompileOutputs="false" prevents compiling the generated files into an assembly.

Note that an empty assembly is still generated, but you should ignore it. As with any build system, it is used to detect out-of-date dependencies and recompile them.

I am getting a warning about a missing expected file!

When we are preparing compile, there is no way to know whether a given proto file will produce a *Grpc.cs output or not. If the proto file has a service clause, it will; otherwise, it won't, but the build script cannot know that in advance. When we are treating generated .cs files as temporary, this is ok, but when generating them for you, creating empty files is probably not. You need to tell the compiler which files should be compiled with gRPC services, and which only contain protobuffer message definitions.

One option is just ignore the warning. Another is quench it by setting the property Protobuf_NoWarnMissingExpected to true:

<PropertyGroup>
  <Protobuf_NoWarnMissingExpected>true</Protobuf_NoWarnMissingExpected>
</PropertyGroup>

For a small to medium projects this is sufficient. But because of a missing output dependency, the corresponding .proto file will be recompiled on every build. If your project is large, or if other large builds depend on generated files, and are also needlessly recompiled, you'll want to prevent these rebuilds when files have not in fact changed, as follows:

Explicitly tell protoc for which files it should use the gRPC plugin

You need to set the Protobuf item property GrpcServices to None for those .proto inputs which do not have a service declared (or, optionally, those which do but you do not want a service/client stub for). The default value for the GrpcServices is Both (both client and server stub are generated). This is easy enough to do with glob patterns if your files are laid out in directories according to their service use, for example:

  <ItemGroup>
    <Protobuf Include="**/*.proto" OutputDir="%(RelativePath)"
              CompileOutputs="false" GrpcServices="None" />
    <Protobuf Update="**/hello/*.proto;**/bye/*.proto" GrpcServices="Both" />
  </ItemGroup>

In this sample, all .proto files are compiled with GrpcServices="None", except for .proto files in subdirectories on any tree level named hello/ and bye, which will take GrpcServices="Both" Note the use of the Update attribute instead of Include. If you write Include by mistake, the files will be added to compile twice, once with, and once without GrpcServices. Pay attention not to do that!

Another example would be the use of globbing if your service .proto files are named according to a pattern, for example *_services.proto. In this case, The Update attribute can be written as Update="**/*_service.proto", to set the attribute GrpcServices="Both" only on these files.

But what if no patterns work, and you cannot sort a large set of .proto file into those containing a service and those not? As a last resort,

Force creating empty .cs files for missing outputs.

Naturally, this results in a dirtier compiler output tree, but you may clean it using other ways (for example, by not copying zero-length .cs files to their final destination). Remember, though, that the files are still important to keep in their output locations to prevent needless recompilation. You may force generating empty files by setting the property Protobuf_TouchMissingExpected to true:

  <PropertyGroup>
    <Protobuf_TouchMissingExpected>true</Protobuf_TouchMissingExpected>
  </PropertyGroup>

But I do not use gRPC at all, I need only protobuffer messages compiled

Set GrpcServices="None" on all proto files:

  <ItemGroup>
    <Protobuf Include="**/*.proto" OutputDir="%(RelativeDir)"
              CompileOutputs="false" GrpcServices="None" />
  </ItemGroup>

That's good so far, but I do not want the bin and obj directories in my tree

You may create the project in a subdirectory of the root of your files, such as, for example, .build. In this case, you want to refer to the proto files relative to that .build/ directory as

  <ItemGroup>
    <Protobuf Include="../**/*.proto" ProtoRoot=".."
              OutputDir="%(RelativeDir)" CompileOutputs="false" />
  </ItemGroup>

Pay attention to the ProtoRoot property. It needs to be set to the directory where import declarations in the .proto files are looking for files, since the project root is no longer the same as the proto root.

Alternatively, you may place the project in a directory above your proto root, and refer to the files with a subdirectory name:

  <ItemGroup>
    <Protobuf Include="proto_root/**/*.proto" ProtoRoot="proto_root"
              OutputDir="%(RelativeDir)" CompileOutputs="false" />
  </ItemGroup>

Alas, this all is nice, but my scenario is more complex, -OR-

I'll investigate that when I have time. I just want to run protoc as I did before.

One option is examine our .targets and .props files and see if you can create your own build sequence from the provided targets so that it fits your needs. Also please open an issue (and tag @kkm000 in it!) with your scenario. We'll try to support it if it appears general enough.

But if you just want to run protoc using MsBuild <Exec> task, as you probably did before the version 1.17 of Grpc.Tools, we have a few build variables that point to resolved names of tools and common protoc imports. You'll have to roll your own dependency checking (or go with a full recompilation each time, if that works for you), but at the very least each version of the Tools package will point to the correct location of the files, and resolve the compiler and plugin executables appropriate for the host system. These property variables are:

  • Protobuf_ProtocFullPath points to the full path and filename of protoc executable, e. g., "C:\Users\kkm.nuget\packages\grpc.tools\1.17.0\build\native\bin\windows\protoc.exe".

  • gRPC_PluginFullPath points to the full path and filename of gRPC plugin, such as "C:\Users\kkm.nuget\packages\grpc.tools\1.17.0\build\native\bin\windows\grpc_csharp_plugin.exe"

  • Protobuf_StandardImportsPath points to the standard proto import directory, for example, "C:\Users\kkm.nuget\packages\grpc.tools\1.17.0\build\native\include". This is the directory where a declaration such as import "google/protobuf/wrappers.proto"; in a proto file would find its target.

Use MSBuild property expansion syntax $(VariableName) in your protoc command line to substitute these variables, for instance,

  <Target Name="MyProtoCompile">
    <PropertyGroup>
      <ProtoCCommand>$(Protobuf_ProtocFullPath) --plugin=protoc-gen-grpc=$(gRPC_PluginFullPath)  -I $(Protobuf_StandardImportsPath) ....rest of your command.... </ProtoCCommand>
    </PropertyGroup>
    <Message Importance="high" Text="$(ProtoCCommand)" />
    <Exec Command="$(ProtoCCommand)" />
  </Target>

Also make sure not to include any file names to the Protobuf item collection, otherwise they will be compiled by default. If, by any chance, you used that name for your build scripting, you must rename it.

What about C++ projects?

This is in the works. Currently, the same variables as above are set to point to the protoc binary, C++ gRPC plugin and the standard imports, but nothing else. Do not use the Protobuf item collection name so that your project remains future-proof. We'll use it for C++ projects too.

Reference

Protobuf item metadata reference

The following metadata are recognized on the <Protobuf> items.

Name Default Value Synopsis
Access public public, internal Generated class access
ProtoCompile true true, false Pass files to protoc?
ProtoRoot See notes A directory Common root for set of files
CompileOutputs true true, false C#-compile generated files?
OutputDir See notes A directory Directory for generated C# files
GrpcOutputDir See notes A directory Directory for generated stubs
GrpcServices both none, client, Generated gRPC stubs
server, both

Notes

  • ProtoRoot
    For files inside the project cone, ProtoRoot is set by default to the project directory. For every file outside of the project directory, the value is set to this file's containing directory name, individually per file. If you include a subtree of proto files that lies outside of the project directory, you need to set this metadatum. There is an example in this file above. The path in this variable is relative to the project directory.

  • OutputDir
    The default value for this metadatum is the value of the property Protobuf_OutputPath. This property, in turn, unless you set it in your project, will be set to the value of the standard MSBuild property IntermediateOutputPath, which points to the location of compilation object outputs, such as "obj/Release/netstandard1.5/". The path in this property is considered relative to the project directory.

  • GrpcOutputDir
    Unless explicitly set, will follow OutputDir for any given file.

  • Access
    Sets generated class access on both generated message and gRPC stub classes.

grpc_csharp_plugin command line options

Under the hood, the Grpc.Tools build integration invokes the protoc and grpc_csharp_plugin binaries to perform code generation. Here is an overview of the available grpc_csharp_plugin options:

Name Default Synopsis
no_client off Don't generate the client stub
no_server off Don't generate the server-side stub
internal_access off Generate classes with "internal" visibility
lite_client off Generate client stubs that inherit from "LiteClientBase"

Note that the protocol buffer compiler has a special commandline syntax for plugin options. Example:

protoc --plugin=protoc-gen-grpc=grpc_csharp_plugin --csharp_out=OUT_DIR \
    --grpc_out=OUT_DIR --grpc_opt=lite_client,no_server \
    -I INCLUDE_DIR foo.proto