* Use latest pylint in Python 3.7 (they dropped support for PY2)
* Make latest pylint happy
* Forced to upgrade to shellcheck 0.4.4
* Make shellcheck 0.4.4 happy
* Adopt reviewers' advice to reduce global disabled rules
This addresses https://github.com/grpc/grpc/issues/17001. Prior to
https://github.com/grpc/grpc/pull/13603, our credentials cython objects
used grpc_initi() and grpc_shutdown() on creation and destruction. These are
now managed differently, but the grpc_init() and grpc_shutdown() calls
are still required. See the MetadataCredentialsPluginWrapper in C++,
which extends the GrpcLibraryCodegen class to ensure that grpc_init()
and grpc_shutdown() are called appropriately.
Without this, we can deadlock when a call to grpc.Channel#close()
triggers grpc_shutdown() to block and wait for all timer threads to
finish: one of these timer threads may end up unreffing the subchannel
and triggering grpc_call_credentials_unref, which will jump back into
Cython and hang when it tries to reacquire the GIL.
* Expose the C-Core API in Cython layer
* Handle the object translation
* Create a separate package for Channelz specifically
* Handle nullptr and raise exception if seen one
* Translate C++ Channelz unit tests
* Adding 5 more invalid query unit tests
Adding peripheral utility for grpcio-channelz package
* Add to `pylint_code.sh`
* Add to Python build script
* Add to artifact build script
* Add to Bazel
* Add to Sphinx module list
Beta code elements are going to get deprecated and
Bazel support is much newer, so Bazel users are
not supposed to accidentally depend on beta code
elements. Preventing Bazel from building and
including beta code elements makes our tests pass
without depending on beta in grpcio target and
helps avoid including that dependency accidentally
if you are using Bazel.
This commit resolves an interop test currently failing on master. Over
the past couple of weeks, bazel-based tests have been introduced. The
current setup does not appear to automatically handle transitive
dependencies. Instead, transitive dependencies such as `requests` have
been manually added to `requirements.bazel.txt`. It appears that the
build server happened to have the dependencies of the `requests` library
installed already, but later had a configuration wipe.
This was compounded by the google-auth library erroneously reporting
that the `requests` module itself was not installed. In fact, it was the
transitive dependencies of `requests` that were not being installed by
the build file. (third-order dependencies of our test)
I consider this a quick fix to get the build passing. In the long run,
we need to automatically resolve and install transitive dependencies in
our bazel builds.
Resolves: #17170
I was trying to get a feel for what the rest of the python ecosystem
does with its logging, so I looked into the top few libraries on pypi:
urllib3 maintains a logger for not quite every module, but for each
one that does heavy lifting. The logger name is `__name__`, and no
handlers are registered for any module-level loggers, including
NullHandler. Their documentation spells out how to configure logging
for the library.
They explicitly configure a library root-level logger called `urllib3`
to which they attach a `NullHandler`. This addresses the "no handlers
could be found" problem.
Their tests explicitly configure handlers, just like ours do.
scrapy is more hands-on. It provides a configuration module for its
logging and a whole document on how to handle logging with scrapy. It
looks like log.py's whole reason for existence is making sure that a
handler is attached to to the scrapy handler at startup.
I think the extra complexity here is because scrapy also offers a CLI,
so there has to be some way to configure logging without resorting to
writing python, so I felt we didn't need to resort to this added
complexity.
---
Based on all of the libraries I've looked at, I think our current
approach is reasonable. The one change I would make is to explicitly
configure a `grpc` logger and to only attach a `NullHandler` to it
instead of putting the burden on the author of each new module to
configure it there.
With this change, I have
- Configured a logger in each module that cares about logging
- Removed all NullHandlers attached to module-level loggers
- Explicitly configured a `grpc` logger with a `NullHandler` attached
Resolves: #16572
Related: #17064
* Use templates instead of generating them every time
* Theme changed
* Add grpc_* modules
* APIs grouped
* No documentation for class members without docstring
* Add docstring for status code
* Include unit tests to test default behaviour, disable behaviour, enable behaviour of the wait-for-ready mechanism
* Import flags constants from grpc_types.h
* Use WaitGroup to wait for TRANSIENT_FAILURE state in unit test
Add interop tests for gRPC Python. py_proto_library rules are added to
src/proto/grpc/testing/BUILD since grpc_proto_library is not compatible
with py_* rules.
'requests' python module is added to requirements.bazel.txt as it is a
dependency for google-auth. Previously, this was installed through
tools/run_tests/helper_scripts/build_python.sh before running tests.
If the last reference to a Python object is at module scope, when
its destructor is run before program termination it may find that
the globals it requires no longer exist. Destructors of objects
likely to be stored at module global scope need to check that
globals exist before attempting to use them, to avoid warnings
being printed by the Python interpreter.
See grpc#17004
* unit test included
* throw ValueError exception from Cython to Python
* prevent the deconstruction method from failing when Channel initialization failed
Add interop tests for gRPC Python. py_proto_library rules are added to
src/proto/grpc/testing/BUILD since grpc_proto_library is not compatible
with py_* rules.
'requests' python module is added to requirements.bazel.txt as it is a
dependency for google-auth. Previously, this was installed through
tools/run_tests/helper_scripts/build_python.sh before running tests.
Add Bazel rules for building and testing grpcio_health_checking.
An unofficial fork for rules_protobuf is used for now as it incorporates
a change (#196 by duduko on the upstream repo pubref/rules_protobuf)
which allows the protoc compiler to compile generated protos too. This
was not merged because the change was failing for golang, but works as
expected for Python.
This is needed because grpcio_health_checking fetches it's proto file
from a different directory (previously achived through setup.py) and
thus needs to be moved to the required location within bazel-genfiles
using a genrule.
There are exactly 50 tests under grpcio_tests/tests/testing and
grpcio_tests/tests/unit for gRPC Python. Add Bazel targets for 44 of
them. unit._reconnect_test has been left out for now due to an
unexpected RuntimeError (#16336) and unit._server_ssl_cert_config_test
because of changes needed to the import path within
src/python/grpcio_tests/tests/testing/proto/services.proto (will
possibly be included as a fix to #15370).
The count of tests here is reported the way unittest counts tests, which
is the number of individual unit test classes it sees. This will be
different from Bazel's count, which counts the number of Bazel test
targets, which (by convention mostly) is one per test module.
A process may fork after invoking grpc_init() and use gRPC in the child
if and only if the child process first destroys all gRPC resources
inherited from the parent process and invokes grpc_shutdown().
Subsequent to this, the child will be able to re-initialize and use
gRPC. After fork, the parent process will be able to continue to use
existing gRPC resources such as channels and calls without interference
from the child process.
To facilitate gRPC Python applications meeting the above constraints,
gRPC Python will automatically destroy and shutdown all gRPC Core
resources in the child's post-fork handler, including cancelling
in-flight calls (see detailed design below). From the client's
perspective, the child process is now free to create new channels and
use gRPC.
Module level loggers were introduced to gRPC Python in 06e1683, but
missed configuring these, leading to 'No handler found for module'
errors. Using the root logger implicitly calls basicConfig() which does
the basic configuration for the logging system by creating a
StreamHandler with a default Formatter and adding it to the logger. But
this is not the case for module level loggers.
Fix this issue by explicitly calling logging.basicConfig().
Added a flag-guarded feature that allows gRPC to load TLS/SSL
roots from the OS trust store. This is the Linux-specific
implementation of such feature.
Change the blocking stream-unary call code path to rely
on the underlying synchronous API, as opposed to calling
the Future-based underlying async API and invoking `.result()`
on the returned Future object immediately, which can be
resource-intensive.
Change the blocking unary-unary call code path to rely
on the underlying synchronous API, as opposed to calling
the Future-based underlying async API and invoking `.result()`
on the returned Future object immediately, which can be
resource-intensive.
All files under `grpc/_cython/_cygrpc` in grpcio Python package
are used as include files and thus have a .pxi file extension.
grpc_gevent implementation was added in 1bfff8e, but didn't include the
.pxi file extension. Update these file names.
This change adds an experimental ssl_session_cache_lru function to the
Python API that returns an encapsulated grpc_ssl_session_cache (#14483).
Python clients may use this object as an argument for the
grpc.ssl_session_cache channel option if they wish to cache and resume
TLS sessions with a server.
@nathanielmanistaatgoogle and I just had a good session helping me
figure out how to use this API correctly, and this rewording of the
documentation was the result.
All logging in Python so far was done with the root logger, resulting
in logs like: `ERROR:Exception calling application:`. With module-level
loggers, the logs will instead include the module in which the
exception is raised: `ERROR:grpc._server:Exception calling application:`
This removes invocation-side completion queues from the _cygrpc API.
Invocation-side calls are changed to no longer share the same lifetime
as Core calls.
Illegal metadata is now detected on invocation rather than at the start
of a batch (so passing illegal metadata to a response-streaming method
will now raise an exception immediately rather than later on when
attempting to read the first response message).
It is no longer possible to create a call without immediately starting
at least one batch of operations on it. Only tests are affected by this
change; there are no real use cases in which one wants to start a call
but wait a little while before learning that the server has rejected
it.
It is now required that code above cygrpc.Channel spend threads on
next_event whenever events are pending. A cygrpc.Channel.close method
is introduced, but it merely blocks until the cygrpc.Channel's
completion queues are drained; it does not itself drain them.
Noteworthy here is that we drop the cygrpc.Channel.__dealloc__ method.
It is not the same as __del__ (which is not something that can be added
to cygrpc.Channel) and there is no guarantee that __dealloc__ will be
called at all or that it will be called while the cygrpc.Channel
instance's Python attributes are intact (in testing, I saw both in
different environments). This commit does not knowingly break any
garbage-collection-based memory management working (or "happening to
appear to work in some circumstances"), though if it does, the proper
remedy is to call grpc.Channel.close... which is the objective towards
which this commit builds.
The Beta API has an execution date and RPC Framework is but a distant
memory.
This test is flaky with Python 3.5 on Windows! Some mysteries will just
have to remain unsolved...
Currently, `warning: sdist: standard file not found` warning is always
printed when creating source distribution for gRPC python reflection
package. This should stop that.
The thread that watches connectivity on the channel might not
pick up that the server has gone away before the request is
dispatched, and return UNAVAILABLE instead of reconnecting
prior to sending the request. The fundamental solution would
basically be enabling retries in C-core. For now, we opt to
sleep a second to deflake this particular test case.
A gRPC Python client interceptor is passed an instance of a class
that implements the ClientCallDetails interface. The interceptor
can choose to create its own object that implements the interface,
and pass it back to the continuation invoked by the interceptor.
To make it easy to add additional attributes to call details,
without breaking user code that hardcode the attributes required
by the interface, instead of interospecting the object passed
to the interceptor at runtime, and to ease authorship of
interceptors that want to keep some attributes intact and not
care about them, we relax the requirements on the object that
is expected to get passed by the interceptor and let the user
omit some attributes. Omitted attributes will be replaced
by the original value of the attribute given to the interceptor.
grpc.ServicerContext.abort is documented to always raise an exception
to terminate the RPC. The code argument "must not be StatusCode.OK."
However, if you do pass StatusCode.OK, the RPC terminates successfully
on the client side, but returns None.
_server.py: If the user accidentally passes StatusCode.OK, treat it as
StatusCode.UNKNOWN. This is what happens if the user accidentally
passes something that is not a StatusCode instance. Additionally
set details to ''.
_metadata_code_details_test.py: update test to verify the behavior of
abort with invalid codes.
Rather than single classes they are now broken up into class families
with each class containing only those fields and methods that are
needed in the context in which the class is used.
Picks up work from https://github.com/grpc/grpc/pull/10259.
A merge was impossible due to the many sweeping changed that
have occured since I last touched that PR (c++-ization, exec_ctx,
reorganitation of filters, etc).
It is now a family of classes conforming to an interface rather than a
single class no single instance of which makes use of all behavior
scoped to the class.
It also now only uses gRPC Core memory for the time of a single batch
rather than for the entire lifetime of the instance.
When we set the call state to "CANCELLED" after
grpc_cancel_all_calls, we would block other start batch
operations from happening. The rpc_state for the cancelled
call would still be in the server's rpc_states set, but it
would never get removed because there were no active batches
for the call, and the only place we remove from rpc_states is
when a batch completes.
It is better to rely on c-core's cancellation. Once a call
is cancelled, all subsequent ops on that call will return
immediately with a cancellation error.
The RLock() change is due to the possibility that
_on_call_completed
gets invoked immediately when the call has already completed when the
rpc_future callback is created.
gRPC Python required RPCs terminating with non-OK status code to still
return a valid response value after calling set_code, even though the
response value was not supposed to be communicated to the client, and
returning None is considered a programming error.
This commit introduces an alternative mechanism to terminate RPCs by
calling the `abort` method on `ServicerContext` passed to the handler,
which raises an exception and signals to the gRPC runtime to abort the
RPC with the specified status code and details.
Richard Belleville