223 lines
11 KiB
Markdown
223 lines
11 KiB
Markdown
# Grpc.Tools MSBuild integration overview
|
|
|
|
This is an overview for maintainers of Grpc.Tools.
|
|
|
|
The Grpc.Tools NuGet package provides custom build targets to make it easier to specify `.proto` files
|
|
in a project and for those files to be compiled and their generated files to be included in the project.
|
|
|
|
# Files in the NuGet package
|
|
|
|
## .props and .target files
|
|
|
|
MSBuild properties and targets included from the Grpc.Tools NuGet package are in:
|
|
|
|
* `build\Grpc.Tools.props`, which imports
|
|
* `build\_grpc\_Grpc.Tools.props`
|
|
* `build\_protobuf\Google.Protobuf.Tools.props`
|
|
* `build\Grpc.Tools.targets`, which imports
|
|
* `build\_grpc\_Grpc.Tools.targets`
|
|
* `build\_protobuf\Google.Protobuf.Tools.targets`
|
|
|
|
Details of how NuGet packages can add custom build targets and properties to a project is documented
|
|
here: [MSBuild .props and .targets in a package](https://learn.microsoft.com/en-us/nuget/concepts/msbuild-props-and-targets)
|
|
|
|
Basically the `.props` and `.targets` files are automatically included in the projects - the `.props` at the top
|
|
of the project and the `.targets` are added to the bottom of the project.
|
|
|
|
## Visual Studio property pages
|
|
|
|
For Visual Studio integration - these files provide the properties pages:
|
|
|
|
* `build\_protobuf\Protobuf.CSharp.xml` (included from `Google.Protobuf.Tools.targets`)
|
|
* `build\_grpc\Grpc.CSharp.xml` (included from `_Grpc.Tools.targets`)
|
|
|
|
## Custom tasks DLLs
|
|
|
|
DLLs containing the custom tasks are in:
|
|
|
|
* `build\_protobuf\netstandard1.3`
|
|
* `build\_protobuf\net45`
|
|
|
|
## Protobuf compiler and C# gRPC plugin binaries
|
|
|
|
Native binary executables for the protobuf compiler (_protoc_) and C# gRPC plugin (_grpc_csharp_plugin_) are
|
|
included in the NuGet package. Included are binaries for various OSes (Windows, Linux, macOS) and
|
|
CPU architectures (x86, x64, arm64).
|
|
|
|
The build determines which executables to use for the particular machine that the it is being run on.
|
|
These can be overridden by specifying MSBuild properties or environment variables to give the paths to custom executables:
|
|
|
|
* `Protobuf_ProtocFullPath` property or `PROTOBUF_PROTOC` environment variable \
|
|
Full path of protoc executable
|
|
* `gRPC_PluginFullPath` property or `GRPC_PROTOC_PLUGIN` environment variable \
|
|
Full path of gRPC C# plugin
|
|
|
|
# Grpc.Tools custom build targets
|
|
|
|
## Hooking the custom targets into the project build
|
|
|
|
The custom targets hook into various places in a normal MSBuild build by specifying
|
|
before/after targets at the relevant places. See
|
|
[msbuild-targets](https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-targets)
|
|
for predefined targets.
|
|
|
|
* Before `PrepareForBuild`
|
|
* we add `Protobuf_SanityCheck` that checks this is a supported project type, e.g. a C# project.
|
|
* Before `BeforeCompile`
|
|
* we add all the targets that compile the `.proto` files and generate the expected `.cs` files. These files are added to those that get compiled by the C# compiler.
|
|
* The target `_Protobuf_Compile_BeforeCsCompile` is the _glue_ inserting the targets into the build.
|
|
It may look like it isn't doing anything but by specifying `BeforeTargets` and `DependsOnTargets` it inserts `Protobuf_Compile` into the build - but only doing so if this is a C# project.
|
|
* After `CoreClean`
|
|
* we add `Protobuf_Clean` that cleans the files generated by the protobuf compiler.
|
|
* The target `_Protobuf_Clean_AfterCsClean` is the _glue_ inserting `Protobuf_Clean` into the build - but only doing so if this is a C# project.
|
|
|
|
## Custom tasks
|
|
|
|
There are a few custom tasks needed by the targets. These are implemented in C# in the Grpc.Tools project
|
|
and packaged in the file `Protobuf.MSBuild.dll` in the NuGet package. See [task writing](https://learn.microsoft.com/en-us/visualstudio/msbuild/task-writing) for information about implementing custom tasks.
|
|
|
|
* ProtoToolsPlatform
|
|
* Works out the operating system and CPU architecture
|
|
* ProtoCompilerOutputs
|
|
* Tries to work out the names and paths of the files that would be generated by the protobuf compiler and returns these as a list of items
|
|
* Also returns list of items that is the same as the `Protobuf` items passed in with the output directory metadata updated
|
|
* ProtoReadDependencies
|
|
* Read generated files from previously written dependencies file and return as items.
|
|
* ProtoCompile
|
|
* Runs the protobuf compiler for a proto file. The executable to run is specified by the property `Protobuf_ProtocFullPath`
|
|
* To do this it:
|
|
* first writes out a response file containing the parameters for protobuf compiler
|
|
* runs the executable, which generates the `.cs` files and a `.protodep` dependencies file
|
|
* reads the dependencies file to find the files that were generated and these are returned as a list of _items_ that can then be used in the MSBuild targets
|
|
|
|
## Build steps
|
|
|
|
The names of these items and properties are correct at the time of writing this document.
|
|
|
|
High level builds steps:
|
|
|
|
* Prepare list of `.proto` files to compile
|
|
* Makes sure all needed metadata is set for the `<Protobuf>` item, defaulting some values
|
|
* Removes `<Protobuf>` items that no longer exist or are marked as don't compile
|
|
* Handling incremental builds
|
|
* Work out files that need to be created or have changed
|
|
* Compile the `.proto` files
|
|
* Add generated files to the list of files for the C# compiler
|
|
|
|
### Prepare the list of .proto files to compile
|
|
|
|
At various stages of the build copies of the original `<Protobuf>` items are created
|
|
and/or updated to set metadata and to prune out unwanted items.
|
|
|
|
Firstly, the build makes sure `ProtoRoot` metadata is set for all `Protobuf` items.
|
|
A new list of items - `Protobuf_Rooted` - is created from the `Protobuf` items with `ProtoRoot` metadata set:
|
|
|
|
* If `ProtoRoot` already set in the `<Protobuf>` item in the project file then it is left as-is.
|
|
* If the `.proto` file is under the project's directory then set `ProtoRoot="."`.
|
|
* If the `.proto` file is outside of the project's directory then set `ProtoRoot="<relative path to project directory>"`.
|
|
|
|
Now prune out from `Protobuf_Rooted` the items that the user doesn't want to compile - those don't have
|
|
`ProtoCompile` metadata as `true`. The pruned list is now called `Protobuf_Compile`.
|
|
|
|
Set the `Source` metadata on `Protobuf_Compile` items to be the name of the `.proto` file.
|
|
The `Source` metadata is used later as a key to map generated files to `.proto` files.
|
|
|
|
### Handling incremental builds
|
|
|
|
#### Gathering files to check for incremental builds
|
|
|
|
The target `Protobuf_PrepareCompile` tries to work out which files the protobuf compiler will
|
|
generate without actually calling the protobuf compiler. This is a best-effort guess.
|
|
The custom task `ProtoCompilerOutputs` is called to do this. The results are stored in the
|
|
item list `Protobuf_ExpectedOutputs`.
|
|
|
|
The target `Protobuf_PrepareCompile` also reads previously written `.protodep` files to get
|
|
any actual files previously generated. The custom task `ProtoReadDependencies` is called to
|
|
do this. The results are stored in the item list `Protobuf_Dependencies`.
|
|
This is in case the list of actual files is different from the previous best-effort guess
|
|
from `ProtoCompilerOutputs`.
|
|
|
|
The expected outputs and previous outputs are needed so that the timestamps of those files
|
|
can be checked later when handling an incremental build.
|
|
|
|
#### Understanding incremental builds
|
|
|
|
To avoid unnecessarily recompiling the `.proto` files during an incremental build the
|
|
target `_Protobuf_GatherStaleBatched` tries to work out if any files have changed.
|
|
|
|
It checks for out of date files using MSBuilds incremental build feature that compares the
|
|
timestamps on a target's _Input_ files to its _Output_ files.
|
|
See [How to: Build incrementally](https://learn.microsoft.com/en-us/visualstudio/msbuild/how-to-build-incrementally)
|
|
|
|
The _Inputs_ that are checked are:
|
|
* Timestamps of the `.proto` files
|
|
* Timestamps of previously generated files (list of these files read from `.protodep` files)
|
|
* Timestamps of MSBuild project files
|
|
|
|
These are checked against the _Outputs_:
|
|
* Timestamps of the expected generated files
|
|
|
|
[MSBuild target batching](https://learn.microsoft.com/en-us/visualstudio/msbuild/item-metadata-in-target-batching)
|
|
is used to check each `.proto` file against its expected outputs. The batching is done by specifying the `Source`
|
|
metadata in the _Input_. Items where `Source` metadata matches in both input and output are in each batch.
|
|
|
|
The target `_Protobuf_GatherStaleBatched` sets the metadata `_Exec=true` on `_Protobuf_OutOfDateProto`
|
|
items that are out of date.
|
|
|
|
Later in the target `_Protobuf_GatherStaleFiles`, the items in `_Protobuf_OutOfDateProto` that don't have
|
|
metadata `_Exec==true` are removed from the list of items, leaving only those that need compiling.
|
|
|
|
### Compile the .proto files
|
|
|
|
The target `_Protobuf_CoreCompile` is run for each `.proto` file that needs compiling.
|
|
These are in the item list `_Protobuf_OutOfDateProto`. The custom task `ProtoCompile` is called to run the
|
|
protobuf compiler. The files that were generated are returned in the item list `_Protobuf_GeneratedFiles`.
|
|
|
|
If there are expected files that were not actually generated then the behaviour depends on whether the
|
|
generated files should have been within the project (e.g. in the intermediate directories) or were
|
|
specified to be outside of the project.
|
|
* If within the project - empty files are created to prevent incremental builds doing unnecessary recompiles
|
|
* If outside the project - by default empty files are not created and a warning is output (this behaviour is configurable)
|
|
|
|
**TODO:** why are files inside and outside the project treated differently?
|
|
|
|
### Add generated .cs files to the list of files for the C# compiler
|
|
|
|
The target `_Protobuf_AugmentLanguageCompile` adds to the `Compile` item list
|
|
(the list of files that CSC compiles) the expected generated files.
|
|
|
|
**Note** - this is the _expected_ files not the _actual_ generated files and this is done
|
|
before the protobuf compiler is called.
|
|
|
|
**TODO:** why are the _expected_ files not the _actual_ generated files added?
|
|
|
|
## Handling design time builds
|
|
|
|
Design-time builds are special builds that Visual Studio uses to gather information about the project.
|
|
They are not user-initiated but may be triggered whenever files are added, removed or saved.
|
|
See [Design-Time Builds](https://github.com/dotnet/project-system/blob/main/docs/design-time-builds.md).
|
|
|
|
The Grpc.Tools build targets used to try and optimise design time builds by disabling calling the
|
|
protobuf compiler during a design time build. However this optimisation can lead to errors in
|
|
Visual Studio because the generated files may not exist or be out of date and any code that relies
|
|
on them will then have errors.
|
|
|
|
Now design time builds behave exactly the same as a normal build.
|
|
The old behaviour can be enabled by setting the setting `DisableProtobufDesignTimeBuild` property
|
|
to `true` in the project file **_if_** it is a design time build, e.g. by adding
|
|
|
|
```xml
|
|
<PropertyGroup Condition="'$(DesignTimeBuild)' == 'true' ">
|
|
<DisableProtobufDesignTimeBuild>true</DisableProtobufDesignTimeBuild>
|
|
</PropertyGroup>
|
|
```
|
|
|
|
## Automatically including .proto files
|
|
|
|
For SDK projects it is possible to automatically include `.proto` files found in the project
|
|
directory or sub-directories, without having to specify them with a `<Protobuf>` item.
|
|
To do this the property `EnableDefaultProtobufItems` has be set to `true` in the project file.
|
|
|
|
By default it is not set and `<Protobuf>` items must be included in the project for
|
|
the `.proto` files to be compiled.
|