* Enable logs for error message in call_op_set.h.
The `gpr_log` was commented out from call_op_set.h at PR: https://github.com/grpc/grpc/pull/20681, because of the undefined error. It is probably just missing the header file, so adding the `gpr_log` back.
* Automated change: Fix sanity tests (#31178)
Co-authored-by: nanahpang <nanahpang@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: nanahpang <nanahpang@users.noreply.github.com>
* Reland x2: Make GetDefaultEventEngine return a shared_ptr
* remove thread leak from NativeDNSResolver
This is not going to work for resolvers that support cancellation.
* give resolvers bounded lifetimes
Some resolver own EventEngines. EventEngines cannot run off the end of
the process since they have unjoined threads (problematic in a small set
of environments). This gives resolvers bounded lifetimes, and allows
replacement of resolvers without ASAN issues of deleting resolvers in
active use (occurs in tests).
* fix
* fix windows
* fix surface init test
* fix
* sanitize
* use after move
* the test must wait for the callback to be destroyed
* windows fix: delete the resolver on iomgr shutdown, not before
* Make TimerManager threads non-joinable
On gRPC shutdown, any unjoined TimerManager threads will cause TSAN to
detect thread leaks. This fix resolves issues I saw in end2end test
shutdown in another PR, where a single timer manager thread was always
alive after the test ended.
The long-term solution is to integrate the new ThreadPool here, but this
unblocks me for now.
* backport fix
* fix
* shared_ptr<EventEngine> in EventEngine benchmarks
This is an API change required for subsequent work in using EventEngine
shared_ptrs. It has been landed and reverted with other work multiple
times, so I'm landing it independently.
* Add a provision to allow specification of separate set of channel args for the grpclb channel
* fix asan issue
* review comments
* review comments
* add missing file
* remove unused hdr
* fix sanity
* fix comments
* remove unused hdr
* Reland: "Make GetDefaultEventEngine return a shared_ptr (#30280)"
This reverts commit 45959e7cc1.
* Attempted fix with NoDestruct
* Not a process-wide singleton for the type. Just a NonDestruct
* fix
This works around valgrind memory leaks by giving EventEngines a fixed
lifetime. We eventually want ref-counted EventEngines internally, so this is
a step in the right direction as well.
* [filterstack] Eliminate flags on initial metadata
Instead prefer to pass this data in non-encoded metadata.
Needed to fix MetadataMap::Copy to copy non-encoded metadata (not doing so was unintended).
Removed the flags from call_details because there's not been a use for them there for a very long time.
* Automated change: Fix sanity tests
* fix
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
This functionality is entirely implemented in the C++ bindings layer, and so does not need a flag passed down to core (which core already ignores silently).
This provides a move-only alternative to std::function.
fix oracle and engine factory
fix generate_projects for any_invocable
fix
format & iwyu
Automated change: Fix sanity tests (#30205)
Co-authored-by: drfloob <drfloob@users.noreply.github.com>
Co-authored-by: drfloob <drfloob@users.noreply.github.com>
This commit adds a new overloaded version of `CreateBinderChannel`, that accepts
an intent URI (which should be able to be parsed by
https://developer.android.com/reference/android/content/Intent#parseUri(java.lang.String,%20int) )
for specifying the component to connect to.
Later we will deprecate the old APIs that accepts `package_name`,
`class_name`, and `action_name` separately. Intent URI seems to be a
better and more flexible way to specify a component to connect to.
`grpc.binder.custom_android_intent_action_name` channel arg is deprecated now.
User should use URI to specify custom action instead.
* [WIP] Precondition ChannelArgs with a default EventEngine
This is a step towards using ChannelArgs as the primary means of
accessing EventEngine instances in gRPC-core. If not explicitly provided
by the application, a default EventEngine will populated into
ChannelArgs during preconditioning.
This is not a final state, we may want to enable ref-counting here
instead of using raw pointers. And a refactoring is in order to enable
GetObject instead of the more verbose
GetPointer<EventEngine>(GRPC_ARG_EVENT_ENGINE).
* Refactor ChannelArgs::GetObject to support non-conforming classes
This allows us to not expose `ChannelArgName` in the public interface.
* Add std::shared_ptr to ChannelArgs; Add EventEngine specialization
* subchannel fix; cleanup
* replace GetSharedPtr with overloads of GetObjectRef
* Automated change: Fix sanity tests
* fix the fixer
* fix raw pointer retrieval from stored shared_ptr
* Make GetObjectRef<EventEngine> work (not general to shared_ptr)
* enable shared_ptr ChannelArg support for shared_from_this
* use new EventEngines for tests (not the default global)
* Automated change: Fix sanity tests
Co-authored-by: drfloob <drfloob@users.noreply.github.com>
* EventEngine::RunAt - Subchannel connection retries
* Fix refcounting on retry timer reset
* Automated change: Fix sanity tests
* fix autofixer's goof
* Squashed commit of the following:
commit d4aed9e615
Author: Craig Tiller <ctiller@google.com>
Date: Fri May 20 12:58:47 2022 -0700
Revert "[c++14] Remove Capture type (#29327)" (#29748)
This reverts commit 944c0b2ce9.
commit 965feb5726
Author: apolcyn <apolcyn@google.com>
Date: Fri May 20 12:45:33 2022 -0700
xds: Remove aggregate and logical dns clusters env var guard (#29742)
* Remove aggregate and logical dns clusters env var guard
commit d5c8bbce51
Author: Sergii Tkachenko <sergiitk@google.com>
Date: Fri May 20 10:44:59 2022 -0700
xds-k8s: Do not recommend enabling mesh certs by default (#29743)
This should covered separately per this note:
> For more details, and for the setup for security tests, see
["Setting up Traffic Director service security with proxyless gRPC"](https://cloud.google.com/traffic-director/docs/security-proxyless-setup)
user guide.
commit 1df32ca680
Author: AJ Heller <hork@google.com>
Date: Fri May 20 10:18:53 2022 -0700
Delete the EventEngine-driven iomgr implementation (#29654)
This code is not compiled by default and has fallen out of sync with the
rest of the codebase. There's a good chance it won't be used, given our
current work to use an iomgr-drive EventEngine instead.
This code will continue to live in git history, should we need to bring
pieces of it back.
commit 944c0b2ce9
Author: Craig Tiller <ctiller@google.com>
Date: Fri May 20 09:56:23 2022 -0700
[c++14] Remove Capture type (#29327)
* Remove Capture type
* Automated change: Fix sanity tests
* update
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
commit fd744e081d
Author: Esun Kim <veblush@google.com>
Date: Fri May 20 08:54:33 2022 -0700
Removed manylinux2010 python artifacts (#29734)
* Removed manylinux2010 python artifacts
* Fix
* Fix2
* Added cp37 to presubmit test for distribtest relying on cp37 artifacts
commit 5051566b27
Author: Jan Tattermusch <jtattermusch@google.com>
Date: Fri May 20 11:07:04 2022 +0200
Enable remote cache for selected ObjC bazel tests. (#29731)
* enable remote cache for ObjC bazel tests
* add bazel RBE cache for mac ios bazel builds
* release lock before unreffing
* rm some manual WeakRef-counting
* comments
* verbiage
Co-authored-by: drfloob <drfloob@users.noreply.github.com>
Co-authored-by: Vignesh Babu <vigneshbabu@google.com>
* server: add method to expose authority seen by server
* Automated change: Fix sanity tests
Co-authored-by: markdroth <markdroth@users.noreply.github.com>
This code is not compiled by default and has fallen out of sync with the
rest of the codebase. There's a good chance it won't be used, given our
current work to use an iomgr-drive EventEngine instead.
This code will continue to live in git history, should we need to bring
pieces of it back.
* Event engine endpoint read and write API changes
* updating api change
* set google_specific to nullptr by default
* Event-Engine --> EventEngine
* life-time --> lifetime
* removing default args to make sanity checks pass
* Add DNS Server address override to the EventEngine API
* instead, override authority in GetDNSResolver; other fixes
* feedback
* constrain dns_server format to "IP:port"
* Modifying slice buffer add to merge two contiguous slices sharing the same refcount object
* sanity checks
* updating test
* fixing grpc_slice_buffer_add API misuse in proto utils
* fix sanity checks
* minor fix
* Revert "Revert "ORCA: implement ORCA RPC service for OOB backend metric reporting (#29215)" (#29351)"
This reverts commit 71b355624f.
* move ORCA service to its own BUILD rule
* ORCA: implement ORCA RPC service for OOB backend metric reporting
* fix unused parameter error
* gen_upb_api
* add missing build deps
* increase test timing fudge factor
* add missing copyright header
* buildifier
* don't register as a generic service
* report interval defaults to min interval
* don't regenerate the response proto unless something changed
* use INTERNAL for proto parsing failure
* use absl::Duration in public API
Based on a handful of https://abseil.io/tips, it's generally advised to
only fully-qualify namespaces when in a `using` statement, or when it's
otherwise required for compilation. In all other cases, the general
recommendation is to not fully-qualify.
This change fixes most `grpc.*` namespace uses. There are potential
challenges in trying to make blanket changes to non-gRPC namespace uses,
such as `::testing`, since there is also a `grpc::testing` namespace.
Based on a handful of https://abseil.io/tips, it's generally advised to
only fully-qualify namespaces when in a `using` statement, or when it's
otherwise required for compilation. In all other cases, the general
recommendation is to not fully-qualify.
This change fixes most `grpc.*` namespace uses. There are potential
challenges in trying to make blanket changes to non-gRPC namespace uses,
such as `::testing`, since there is also a `grpc::testing` namespace.
We previously have this restriction because some code does not compile
with older NDK version. Now the binder transport implementation loads
the library and resolves symbols at runtime so we don't need the
compile time restriction anymore (and in some case it is overly
restrictive)
Later we might consider remove the GPR_SUPPORT_BINDER_TRANSPORT symbol
and simply use GPR_ANDROID, once we become sure that binder transport
will always be available on Android at compile time.
Partially collapse `event_engine_factory.cc` into `event_engine.cc`. Add a
new function `DefaultEventEngineFactory` which is used to set a default
event engine factory at link time, separate from the factory that can be
set at run time. Implemenet this function in
`default_event_engine_factory.cc`.
This allows alternative default event engine factories to be implemented
without requiring the duplication of the implementations of
`SetDefaultEventEngineFactory`, `CreateEventEngine`, and
`GetDefaultEventEngine`.
* Adding TLS Key export logic to core and c++ wrappers
* Adding and end2end cpp tls key export test and updating broken test due to interface changes
* regenerate projects
* updating tls key export core logic with addition of APIs to grpc_security.h
* undoing changes to tls_security_connector_test
* regenerate projects
* changing the logging format enum name as per GRFC comments
* regenerate projects
* removing some commented code
* updating changes as per review comments
* adding GRPCAPI annotations to functions defined in grpc_security.h
* regenerate projects
* fixed some code styling issues
* removing grpc_security.h include from tls_credentials_options.h
* updating files as per review comments
* minor fixes
* moving some code around
* removing key log format from tls session key log config and converting it to a simple string
* regenerate projects
* fixing mistakes in recent merge with master
* regenerate projects
* regenerate projects
* fixing some distrib and snity errors
* fixing formatting errors
* fixing more sanity checks and raising supported openssl versions to 1.1.1
* updating min supported openssl version to 1.1.1
* updating min supported openssl version in tls_key_export_test
* updating test to fix incorrect vector initialization
* updating as per latest comments
* fixing sanity checks
* addressing review comments
* fixing sanity checks
* fixed c++ comment style
* Automated change: Fix sanity tests
* fixing review comments
Co-authored-by: Vignesh2208 <Vignesh2208@users.noreply.github.com>
Eliminate slice interning, and structures in slices to support it.
Reduces grpc_slice_refcount from 40 bytes (+ a required 8 bytes elsewhere) to 16 bytes.
Removes a pointer dereference for every slice ref/unref.
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* Revert "Revert "[BinderTransport] Allow finding Java class using custom class finder (#28348)" (#28496)"
This reverts commit ecf6a7fa95.
* Add missing include directives
In some some situation it is not feasible to invoke
`InitializeBinderChannelJavaClass` in non-native threads.
And the intended way to find Java class is through class loader cached
at program initialization.
This commit adds a new API that accepts a class finder from user and
uses the function to find and cache binder transport Java util class.
* Plumb subject field and add to authz flow.
* formatting
* Test on empty principal
* resolving comments
* resolving comment
* update subject check in test
* Check if memory owner available prior to polling it
The transport may drop the memory owner during its destruction sequence
* tcp_fix
* Revert "Revert "New resource quota integration (#27643)" (#28014)"
This reverts commit 0ea2c37263.
* clang-format
* fix-path
* fix
* Reintroduce the EventEngine default factory
An application can provide an EventEngine factory function that allows
gRPC internals to create EventEngines as needed. This factory would be
used when no EventEngine is provided for some given channel or server,
and where an EventEngine otherwise could not be provided by the
application. Note that there currently is no API to provide an
EventEngine per channel or per server.
I've also deleted some previous iterations on global EventEngine and
EventEngine factory ideas. This new code lives in a public API, and
coexists with iomgr instead of being isolated to an EventEngine-specific
iomgr implementation.
* add proper namespaces, and fix description
* put factory functions in their own file (for replaceability)
* add synchronization
* generate_projects.sh
* extract event_engine_base and event_engine_factory targets
Also separate iomgr/event_engine files in the BUILD, with comments
* gpr_platform
* move all EE factory declarations to event_engine_base
Makes internal hackery easier.
* add missing deps
* reorder dep alphabetically
* comment style change
* Assert Android API >= v21
This precedes a change that would otherwise break on older Android APIs,
but in a more obvious way.
* error if __ANDROID_API__ is not defined
* update all Andriod minSdkVersions to 21
* csharp experimental: android-19 to 21
Due to limitation of JVM, when user want to create binder channel in
threads created in unmanaged native code, they will need to call this
new API first to make sure Java helper classes can correctly be found.
Unused code in jni_utils.cc are also cleaned up in this commit
* [BinderTransport] Avoid depending on NdkBinder at compile time
We would like to make it possible to use BinderTransport in a APK that
has min sdk version lower than 29 (NdkBinder was introduced at 29)
We copies constants and type definitions from Ndk headers, creates a
same name wrapper for every NdkBinder API we use in
grpc_binder::ndk_util namespace.
We will try to load libbinder_ndk.so and resolve the symbol when the
NdkBinder API wrappers are invoked.
* regenerate projects
* Add GRPC_NO_BINDER guard
* Expose experimental binder transport API
New headers are added
`grpcpp/create_channel_binder.h `: interfaces for creating client
channel
`grpcpp/security/binder_credentials.h`: interfaces for binder server
credentials
`grpcpp/security/binder_security_policy.h`: interfaces for binder
security policy, which is used by both server and client. Individual
security policies are merged into this single header.
Users can now depend on the `grpc++_binder` target to use the headers
listed above.
* Regenerate projects
This introduces the new `Closure` type, and new cancellation semantics.
I've also fixed a few bugs in the EventEngine iomgr implementation that
had gone unnoticed.
* Fix OOM issues in qps tests
* Add more verbose logging.
* Fix clang error
* Fix race between IsCancelled and Read
* Fix build errors from using bool in C code
This commit adds a new symbol GPR_SUPPORT_BINDER_TRANSPORT to
port_platform.h
This will help avoid surprising compilation failure when compiled with
old NDK or low API level
It was determined that an explicit Shutdown method is not necessary, and further, it can be challenging to implement efficiently in some cases. Instead, EventEngines are expected to clean themselves up upon destruction. Anything that relies on an EventEngine's existence must coordinate itself to ensure the engine remains alive for as long as it's needed.
Internal bug b/170934515
In some cases, calling CQ::Next will return true without passing up a
new tag value, which is very much illegal. My expectation is that this
is due to messing up between SHUTDOWN and TIMEOUT in lower layer code
that doesn't particularly matter much to most callers, but was being
erroneously checked here.
It is not possible for such a function to be implemented in a way that
is understood by annotalysis. Mark it deprecated and replace instances
of its use with direct mutex/condvar usage.
Add a bunch of missing thread safety annotations while I'm here.
* Enable retries by default, but add a separate arg for hedging.
* don't need to explicitly enable retries in xDS config selector
* clang-format
* don't need retry_enabled bit anymore
* fix HTTP client filter to restore the send_message op in the batch
* fix retry cancellation when a batch fails on call attempt
* fix clang-tidy
* fix client channel to pass down batches even after cancellation
* fix retry code to pass transport stats back up to the surface
* add some missing payload propagation
* fix retry handling of callbacks for pending batches
* avoid scheduling the same callback twice
* fix some trace messages
* don't avoid starting recv_initial_metadata or recv_message due to recv_trailing_metadata already being started internally
* avoid restarting recv_trailing_metadata after commit if we've already started it internally
* use fast path when retries are not configured
This code adds an iomgr implementation that's backed by an EventEngine. This uses the EventEngine API alone, and separate work will introduce an EventEngine prototype to plug into it.
See also drfloob#1: @nicolasnoble has a pull request against this branch, implementing the libuv-based EventEngine. One goal here is to implement the iomgr code such that it can be merged independently without affecting normal builds.
This implementation can be built using bazel build --cxxopt='-DGRPC_USE_EVENT_ENGINE' :all
Some shortcuts are being taken to get a working, testable version of the engine. EventEngines are not pluggable, for example.
Cheng-Yu Chung