By popular demand, we'll now be offering separate py_grpc_library and
py_proto_library targets sharing the same interface as within google3.
This change necessitated some modifications to how we pull in our own
Python-level dependencies and how we make those available to those
pulling in our project via Bazel.
There is now a grpc_python_deps() Bazel workspace rule that pulls in the
appropriate dependencies, which should be called from the client
project's WORKSPACE file. A test has been added to the bazel/test/
directory to verify that this behavior works as intended.
It's worth noting that the protobuf repository's usage of Starlark
bind() caused a great deal of trouble in ensuring that we could also
pull in six.
This change also required a change in the way generated proto code is
imported in the channelz and health-check modules, as well as in their
associated tests. We were importing them two different ways, each
relative. This resulted in two different module objects being imported
into the process, which were incompatible. I am not sure exactly what
caused this behavior to begin, as this should have been possible before
this PR. As a workaround, I am simply trying two different absolute
imports and using the one that works. This should function both inside
and outside of Bazel environments.
This commit resolves#18331.
This commit resolves#18256.
This commit resolves... another TODO that apparently didn't have an
associated github issue.
We swap out pubref's implementation of py_proto_library with our own,
which more closely mirrors the interface of the internal
py_proto_library, taking the descriptor file output of a proto_library
rule as input.
One minor change in behavior was introduced for simplicity. When a
py_proto_library depends on a proto_library with a source proto file in
a subdirectory of the bazel package, the import module of the resultant
python library will reflect the package, *not* the full directory of the
proto file, including both the bazel package and the subdirectories, as
pubref did previously. This behavior also more closely mirrors google
internal behavior.
This commit also introduces a slightly more stringent bazel format
script. Buildifier on its own will not take care of long lines, but by
running yapf first, we end up with a more legible file. At the moment,
there is no sanity check associated with this formatter.
Currently, `warning: sdist: standard file not found` warning is always
printed when creating source distribution for gRPC python reflection
package. This should stop that.
Using the presence of the `*_pb2_grpc` module, as opposed to the absence
of the build script (`*_commands` module) is a problematic choice,
because even if a generated file is present, the test infrastructure may
want to regenerate it under a different environment (e.g. different
Python/proto package version). This will ensure the protos always get
recompiled if we have a `*_commands` module present, signaling we are in
a build environment, thereby making the process hermetic.
The previous packaging structure exhibited strange behavior of
slowness when trying to use pip to install grpcio-reflection
or grpcio-health-checking in a single line with grpcio-tools.
The root cause seems to be the complicated interaction between
pip and setuptools and the fact that we ship a single .tar.gz
"source" archive for `grpcio_reflection` and
`grpcio_health_checking` packages. `pip` tries to build this
"source" package, and our build process wants to generate
code for the `.proto` files in the package. However, we have
already processed the `.proto` files into `_pb2.py` files in
our artifact build process, and installing `grpcio_tools`
to get `grpcio_{reflection,health_checking}` seems excessive.
The behavior gets worse since `setuptools`, while building
the package from source, tries to fetch `grpcio_tools` from
source and build that too. This takes a while, since it
involves compiling a bunch of native code from `protobuf` and
`grpc` and requires a C compiler to boot.
This commit modifies the Python artifact for the two packages
so that they will not include the raw `.proto` files in the
distribution uploaded to PyPI, nor would they contain the
Python module that does the preprocessing code generation
from the respective .proto files. Instead, a specific code
path is taken when the generated `_pb2_grpc` Python module is
not present in the package to provide such functionality
when built from the gRPC git repository (and hence when built
from our CI infrastructure.)
Alexander Polcyn