* [WIP] Precondition ChannelArgs with a default EventEngine
This is a step towards using ChannelArgs as the primary means of
accessing EventEngine instances in gRPC-core. If not explicitly provided
by the application, a default EventEngine will populated into
ChannelArgs during preconditioning.
This is not a final state, we may want to enable ref-counting here
instead of using raw pointers. And a refactoring is in order to enable
GetObject instead of the more verbose
GetPointer<EventEngine>(GRPC_ARG_EVENT_ENGINE).
* Refactor ChannelArgs::GetObject to support non-conforming classes
This allows us to not expose `ChannelArgName` in the public interface.
* Add std::shared_ptr to ChannelArgs; Add EventEngine specialization
* subchannel fix; cleanup
* replace GetSharedPtr with overloads of GetObjectRef
* Automated change: Fix sanity tests
* fix the fixer
* fix raw pointer retrieval from stored shared_ptr
* Make GetObjectRef<EventEngine> work (not general to shared_ptr)
* enable shared_ptr ChannelArg support for shared_from_this
* use new EventEngines for tests (not the default global)
* Automated change: Fix sanity tests
Co-authored-by: drfloob <drfloob@users.noreply.github.com>
* EventEngine::RunAt - Subchannel connection retries
* Fix refcounting on retry timer reset
* Automated change: Fix sanity tests
* fix autofixer's goof
* Squashed commit of the following:
commit d4aed9e615
Author: Craig Tiller <ctiller@google.com>
Date: Fri May 20 12:58:47 2022 -0700
Revert "[c++14] Remove Capture type (#29327)" (#29748)
This reverts commit 944c0b2ce9.
commit 965feb5726
Author: apolcyn <apolcyn@google.com>
Date: Fri May 20 12:45:33 2022 -0700
xds: Remove aggregate and logical dns clusters env var guard (#29742)
* Remove aggregate and logical dns clusters env var guard
commit d5c8bbce51
Author: Sergii Tkachenko <sergiitk@google.com>
Date: Fri May 20 10:44:59 2022 -0700
xds-k8s: Do not recommend enabling mesh certs by default (#29743)
This should covered separately per this note:
> For more details, and for the setup for security tests, see
["Setting up Traffic Director service security with proxyless gRPC"](https://cloud.google.com/traffic-director/docs/security-proxyless-setup)
user guide.
commit 1df32ca680
Author: AJ Heller <hork@google.com>
Date: Fri May 20 10:18:53 2022 -0700
Delete the EventEngine-driven iomgr implementation (#29654)
This code is not compiled by default and has fallen out of sync with the
rest of the codebase. There's a good chance it won't be used, given our
current work to use an iomgr-drive EventEngine instead.
This code will continue to live in git history, should we need to bring
pieces of it back.
commit 944c0b2ce9
Author: Craig Tiller <ctiller@google.com>
Date: Fri May 20 09:56:23 2022 -0700
[c++14] Remove Capture type (#29327)
* Remove Capture type
* Automated change: Fix sanity tests
* update
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
commit fd744e081d
Author: Esun Kim <veblush@google.com>
Date: Fri May 20 08:54:33 2022 -0700
Removed manylinux2010 python artifacts (#29734)
* Removed manylinux2010 python artifacts
* Fix
* Fix2
* Added cp37 to presubmit test for distribtest relying on cp37 artifacts
commit 5051566b27
Author: Jan Tattermusch <jtattermusch@google.com>
Date: Fri May 20 11:07:04 2022 +0200
Enable remote cache for selected ObjC bazel tests. (#29731)
* enable remote cache for ObjC bazel tests
* add bazel RBE cache for mac ios bazel builds
* release lock before unreffing
* rm some manual WeakRef-counting
* comments
* verbiage
Co-authored-by: drfloob <drfloob@users.noreply.github.com>
Co-authored-by: Vignesh Babu <vigneshbabu@google.com>
* server: add method to expose authority seen by server
* Automated change: Fix sanity tests
Co-authored-by: markdroth <markdroth@users.noreply.github.com>
This code is not compiled by default and has fallen out of sync with the
rest of the codebase. There's a good chance it won't be used, given our
current work to use an iomgr-drive EventEngine instead.
This code will continue to live in git history, should we need to bring
pieces of it back.
* Event engine endpoint read and write API changes
* updating api change
* set google_specific to nullptr by default
* Event-Engine --> EventEngine
* life-time --> lifetime
* removing default args to make sanity checks pass
* Add DNS Server address override to the EventEngine API
* instead, override authority in GetDNSResolver; other fixes
* feedback
* constrain dns_server format to "IP:port"
* Modifying slice buffer add to merge two contiguous slices sharing the same refcount object
* sanity checks
* updating test
* fixing grpc_slice_buffer_add API misuse in proto utils
* fix sanity checks
* minor fix
* Revert "Revert "ORCA: implement ORCA RPC service for OOB backend metric reporting (#29215)" (#29351)"
This reverts commit 71b355624f.
* move ORCA service to its own BUILD rule
* ORCA: implement ORCA RPC service for OOB backend metric reporting
* fix unused parameter error
* gen_upb_api
* add missing build deps
* increase test timing fudge factor
* add missing copyright header
* buildifier
* don't register as a generic service
* report interval defaults to min interval
* don't regenerate the response proto unless something changed
* use INTERNAL for proto parsing failure
* use absl::Duration in public API
Based on a handful of https://abseil.io/tips, it's generally advised to
only fully-qualify namespaces when in a `using` statement, or when it's
otherwise required for compilation. In all other cases, the general
recommendation is to not fully-qualify.
This change fixes most `grpc.*` namespace uses. There are potential
challenges in trying to make blanket changes to non-gRPC namespace uses,
such as `::testing`, since there is also a `grpc::testing` namespace.
Based on a handful of https://abseil.io/tips, it's generally advised to
only fully-qualify namespaces when in a `using` statement, or when it's
otherwise required for compilation. In all other cases, the general
recommendation is to not fully-qualify.
This change fixes most `grpc.*` namespace uses. There are potential
challenges in trying to make blanket changes to non-gRPC namespace uses,
such as `::testing`, since there is also a `grpc::testing` namespace.
We previously have this restriction because some code does not compile
with older NDK version. Now the binder transport implementation loads
the library and resolves symbols at runtime so we don't need the
compile time restriction anymore (and in some case it is overly
restrictive)
Later we might consider remove the GPR_SUPPORT_BINDER_TRANSPORT symbol
and simply use GPR_ANDROID, once we become sure that binder transport
will always be available on Android at compile time.
Partially collapse `event_engine_factory.cc` into `event_engine.cc`. Add a
new function `DefaultEventEngineFactory` which is used to set a default
event engine factory at link time, separate from the factory that can be
set at run time. Implemenet this function in
`default_event_engine_factory.cc`.
This allows alternative default event engine factories to be implemented
without requiring the duplication of the implementations of
`SetDefaultEventEngineFactory`, `CreateEventEngine`, and
`GetDefaultEventEngine`.
* Adding TLS Key export logic to core and c++ wrappers
* Adding and end2end cpp tls key export test and updating broken test due to interface changes
* regenerate projects
* updating tls key export core logic with addition of APIs to grpc_security.h
* undoing changes to tls_security_connector_test
* regenerate projects
* changing the logging format enum name as per GRFC comments
* regenerate projects
* removing some commented code
* updating changes as per review comments
* adding GRPCAPI annotations to functions defined in grpc_security.h
* regenerate projects
* fixed some code styling issues
* removing grpc_security.h include from tls_credentials_options.h
* updating files as per review comments
* minor fixes
* moving some code around
* removing key log format from tls session key log config and converting it to a simple string
* regenerate projects
* fixing mistakes in recent merge with master
* regenerate projects
* regenerate projects
* fixing some distrib and snity errors
* fixing formatting errors
* fixing more sanity checks and raising supported openssl versions to 1.1.1
* updating min supported openssl version to 1.1.1
* updating min supported openssl version in tls_key_export_test
* updating test to fix incorrect vector initialization
* updating as per latest comments
* fixing sanity checks
* addressing review comments
* fixing sanity checks
* fixed c++ comment style
* Automated change: Fix sanity tests
* fixing review comments
Co-authored-by: Vignesh2208 <Vignesh2208@users.noreply.github.com>
Eliminate slice interning, and structures in slices to support it.
Reduces grpc_slice_refcount from 40 bytes (+ a required 8 bytes elsewhere) to 16 bytes.
Removes a pointer dereference for every slice ref/unref.
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* Revert "Revert "[BinderTransport] Allow finding Java class using custom class finder (#28348)" (#28496)"
This reverts commit ecf6a7fa95.
* Add missing include directives
In some some situation it is not feasible to invoke
`InitializeBinderChannelJavaClass` in non-native threads.
And the intended way to find Java class is through class loader cached
at program initialization.
This commit adds a new API that accepts a class finder from user and
uses the function to find and cache binder transport Java util class.
* Plumb subject field and add to authz flow.
* formatting
* Test on empty principal
* resolving comments
* resolving comment
* update subject check in test
* Check if memory owner available prior to polling it
The transport may drop the memory owner during its destruction sequence
* tcp_fix
* Revert "Revert "New resource quota integration (#27643)" (#28014)"
This reverts commit 0ea2c37263.
* clang-format
* fix-path
* fix
* Reintroduce the EventEngine default factory
An application can provide an EventEngine factory function that allows
gRPC internals to create EventEngines as needed. This factory would be
used when no EventEngine is provided for some given channel or server,
and where an EventEngine otherwise could not be provided by the
application. Note that there currently is no API to provide an
EventEngine per channel or per server.
I've also deleted some previous iterations on global EventEngine and
EventEngine factory ideas. This new code lives in a public API, and
coexists with iomgr instead of being isolated to an EventEngine-specific
iomgr implementation.
* add proper namespaces, and fix description
* put factory functions in their own file (for replaceability)
* add synchronization
* generate_projects.sh
* extract event_engine_base and event_engine_factory targets
Also separate iomgr/event_engine files in the BUILD, with comments
* gpr_platform
* move all EE factory declarations to event_engine_base
Makes internal hackery easier.
* add missing deps
* reorder dep alphabetically
* comment style change
* Assert Android API >= v21
This precedes a change that would otherwise break on older Android APIs,
but in a more obvious way.
* error if __ANDROID_API__ is not defined
* update all Andriod minSdkVersions to 21
* csharp experimental: android-19 to 21
Due to limitation of JVM, when user want to create binder channel in
threads created in unmanaged native code, they will need to call this
new API first to make sure Java helper classes can correctly be found.
Unused code in jni_utils.cc are also cleaned up in this commit
* [BinderTransport] Avoid depending on NdkBinder at compile time
We would like to make it possible to use BinderTransport in a APK that
has min sdk version lower than 29 (NdkBinder was introduced at 29)
We copies constants and type definitions from Ndk headers, creates a
same name wrapper for every NdkBinder API we use in
grpc_binder::ndk_util namespace.
We will try to load libbinder_ndk.so and resolve the symbol when the
NdkBinder API wrappers are invoked.
* regenerate projects
* Add GRPC_NO_BINDER guard
* Expose experimental binder transport API
New headers are added
`grpcpp/create_channel_binder.h `: interfaces for creating client
channel
`grpcpp/security/binder_credentials.h`: interfaces for binder server
credentials
`grpcpp/security/binder_security_policy.h`: interfaces for binder
security policy, which is used by both server and client. Individual
security policies are merged into this single header.
Users can now depend on the `grpc++_binder` target to use the headers
listed above.
* Regenerate projects
This introduces the new `Closure` type, and new cancellation semantics.
I've also fixed a few bugs in the EventEngine iomgr implementation that
had gone unnoticed.
* Fix OOM issues in qps tests
* Add more verbose logging.
* Fix clang error
* Fix race between IsCancelled and Read
* Fix build errors from using bool in C code
This commit adds a new symbol GPR_SUPPORT_BINDER_TRANSPORT to
port_platform.h
This will help avoid surprising compilation failure when compiled with
old NDK or low API level
It was determined that an explicit Shutdown method is not necessary, and further, it can be challenging to implement efficiently in some cases. Instead, EventEngines are expected to clean themselves up upon destruction. Anything that relies on an EventEngine's existence must coordinate itself to ensure the engine remains alive for as long as it's needed.
Internal bug b/170934515
In some cases, calling CQ::Next will return true without passing up a
new tag value, which is very much illegal. My expectation is that this
is due to messing up between SHUTDOWN and TIMEOUT in lower layer code
that doesn't particularly matter much to most callers, but was being
erroneously checked here.
It is not possible for such a function to be implemented in a way that
is understood by annotalysis. Mark it deprecated and replace instances
of its use with direct mutex/condvar usage.
Add a bunch of missing thread safety annotations while I'm here.
* Enable retries by default, but add a separate arg for hedging.
* don't need to explicitly enable retries in xDS config selector
* clang-format
* don't need retry_enabled bit anymore
* fix HTTP client filter to restore the send_message op in the batch
* fix retry cancellation when a batch fails on call attempt
* fix clang-tidy
* fix client channel to pass down batches even after cancellation
* fix retry code to pass transport stats back up to the surface
* add some missing payload propagation
* fix retry handling of callbacks for pending batches
* avoid scheduling the same callback twice
* fix some trace messages
* don't avoid starting recv_initial_metadata or recv_message due to recv_trailing_metadata already being started internally
* avoid restarting recv_trailing_metadata after commit if we've already started it internally
* use fast path when retries are not configured
This code adds an iomgr implementation that's backed by an EventEngine. This uses the EventEngine API alone, and separate work will introduce an EventEngine prototype to plug into it.
See also drfloob#1: @nicolasnoble has a pull request against this branch, implementing the libuv-based EventEngine. One goal here is to implement the iomgr code such that it can be merged independently without affecting normal builds.
This implementation can be built using bazel build --cxxopt='-DGRPC_USE_EVENT_ENGINE' :all
Some shortcuts are being taken to get a working, testable version of the engine. EventEngines are not pluggable, for example.
* support user provided audience in gdc and jwt
* fix 1st round of comments
* fix ruby and php to use new GDC API
* fix python clang issue
* address 2nd round of comments
* fix string_view issue
* remove length param in string_view constructor
* De-experimentalize callback API
* Make FromServerContext based on ServerContextBase
* Fix lambda
* Fix headers
* De-experimentalize tests
* clang-format
* Fix consistency checks
* wip
* Fix const-ness of callback client read RPC requests
* Fix golden file
* Give full route_guide callback client example with Hold etc
* Complete example route-guide server
* De-experimentalize test services
* Omit unneeded using
* Remove some uses of non-experimental macro from test
* clang-format examples
* De-experimentalize async stub calls
* Remove experimental namespace use in qps, ubms
* De-experimentalize alarms, generic stubs, and context allocator
* De-experimentalize SetContextAllocator
* clang-format
* Fix conflicts
* Leave obsolete API in place until users can be migrated
* Create client_context_test_peer.h
mock-up class for ClientContext
* Add ClientContextTestPeer as a friend for test
* Update client_context_test_peer.h
* Update client_context_test_peer.h
* unit test for client_context_test_peer.h
* unit test for client_context_test_peer.h
* fix for tools/buildgen/generate_projects.sh
* Revert "fix for tools/buildgen/generate_projects.sh"
This reverts commit b007aa8c2b.
* re-fix for tools/buildgen/generate_projects.sh
* Remove an empty line
Co-authored-by: Vijay Pai <vpai@google.com>
* Revert "Revert "Expose trailers-only response status through C++ callback API (#26249)" (#26363)"
This reverts commit 05c3b30ea0.
* Move the trailers_only check from ClientContext to ClientReactor
* Remove unneeded change in client_context.cc
* Expose trailers-only response status through callback API
* Fix change-detector test
* Address reviewer comment about trailers_only headers
* Add TODO requested by markdroth
* Move variable in struct to improve space efficiency
* Don't instantiate ExecCtx in channel_args destructor unless needed
* Resolve and test race between OnReadDone(ok=false) and IsCancelled
* Fix retry case
* Fix health check case
* Address reviewer comments.
* ADD TODO requested by markdroth
* Adding suffix for stats handling to gRPC
* Fixing typo
* Fixing codegen, adding tests
* Fixing test failures
* Adding nullptr default in StubOptions
* Updating golden test file
* Replacing std::make_unique with absl::make_unique for pre-C++14 compatibility
* Fixing clang format errors
* Reuse stub options for generic stubs
* Cleaning up compiler warnings in GenericStub
* pass XdsClient to LB policies via channel args
* add channel arg for overriding bootstrap config on a per-channel basis
* change tests to use new channel arg approach -- currently failing for server-side tests
* use the same channel args approach on the server side
* clang-format
* fix CircuitBreakingMultipleChannelsShareCallCounter test
* fix XdsEnabledServerTest test
* add TODO
* clang-format
* generate_projects
* fix clang-tidy
* fix build
* attempt to fix python
* Return port number from EventEngine::Listener::Bind
This is an integer, unlike the DNSResolver, since the underlying
bind syscall deals in integers.
* Add return type documentation
* Implement C++ Admin Interface API
* Address reviewer's requests
* Remove static asserts for raw pointers
* Make sanity tests happy
* Windows: pacify conflict between ifndef and macros
* Disable admin services test on iOS
* Make iOS happy by:
* Letting grpcpp_admin conditionally depend on grpcpp_csds
* Fix an unexpected side-effect of dependency update
* Implement the xDS Config Dump as CSDS in Core
* Revemp the logic of caching and constructing the CSDS response
* Unref created error
* Fix the clang tidy complains
* Resolve comments about symbol preload and Json ctor
* Improve readability
* Merge with new ADS logic && support latest CSDS
* Refactor the version/client_status logic
* Add support for error_state
* Add support for does_not_exist
* Add node information
* Fix the ProtoBuf segfault
* More test cases
* Refactor to use bytes as cache instead of JSON
* Apply clang_tidy's suggestion
* Resolve reviewer's requests
* Tiny stylish fixes && make sanity test happy
* WIP: still working on the matchers
* Update a bunch of matchers
* Improve readability a bit
* Resolve reviewer's suggestions
* Resolve reviewer's comments
* Make Unpack fail fast
* Revert "Revert "xDS status notifier (#25321)" (#25702)"
This reverts commit 3c9f3972e3.
* Remove connection from map when OnClose is not registered
* Reviewer comments
* Make annotations real in server header
* More annotations
* clang-fmt
* Annotate shutdown state variables
* Fix annotation of CV predicate
* Fix param list
C-ARES does not work on Fuchsia, and is causing DNS solution failures.
We are disabling C-ARES so that we fall back to the native resolver.
Issue: #25446
Google's internal version of the gRPC codebase uses a version of
protobuf with a different namespace and include paths. This commit makes
parse_json.cc rely on config_protobuf.h for protobuf types so that it
will work properly in that environment. I also replaced a call to
Status::error_code() with a call to Status::code() instead. This way the
code will work with absl::Status, which has a somewhat different API
compared to google::protobuf::util::Status.
AJ Heller