* Reland: "Make GetDefaultEventEngine return a shared_ptr (#30280)"
This reverts commit 45959e7cc1.
* Attempted fix with NoDestruct
* Not a process-wide singleton for the type. Just a NonDestruct
* fix
This works around valgrind memory leaks by giving EventEngines a fixed
lifetime. We eventually want ref-counted EventEngines internally, so this is
a step in the right direction as well.
A (currently) pthread_atfork-based fork support mechanism, allowing EventEngines - or any other object that wants to implement the Forkable interface - respond to forks.
This is a partial fork of the windows iomgr code - specifically the IOCP and Socket pieces - with some improved architecture and encapsulation. And the start of a WindowsEventEngine.
Once this code is used in a gRPC TCP context, I imagine a few issues will shake out. Also, getting sanitizers set up with MSVC will take a bit of work (see a commit referencing abseil and MSVC bugs to hack around).
I forked the IomgrEventEngine's posix poller interfaces in the hope of negotiating compatibility between the platforms, but the interfaces diverged a fair bit, and I'm doubtful we'll be able to use these "pollers" generically in the same TCP code. Reunification might not happen, and that's probably fine, we'll see how similar the TCP code looks once it's fleshed out.
I also extracted the IomgrEventEngine's timer piece into a separate component, usable by both engines.
* [arena] Add ManagedNew(), gtest-ify test
Add a ManagedNew() method to Arena that calls the relevant destructor at Arena destruction time.
There are some cases coming up in the promise based call work where this becomes super convenient, and I expect it's likely that there are other places that's true too.
* Automated change: Fix sanity tests
* review feedback
* use construct/destruct more
* Automated change: Fix sanity tests
* Automated change: Fix sanity tests
* fix
* Automated change: Fix sanity tests
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* Set uses polling to true for oracle_event_engine_posix_test
* add a TODO
* Automated change: Fix sanity tests
Co-authored-by: Vignesh2208 <Vignesh2208@users.noreply.github.com>
* Revert "Revert "Creating a posix oracle event engine and a suite of event engine client tests (#29714)" (#30042)"
This reverts commit 1630efd8ab.
* fix typos
* Creating a posix oracle event engine and a suite of event engine client tests
* regenerate projects
* addressing review feedback
* creating a promise.h in src/core/lib/event_engine
* regenerate projects
* fix errors
* remove no_mac tag for posix oracle event engine test
* Automated change: Fix sanity tests
* fix a static_cast
* adding a no_windows tag
* fixing macos build and test issues
* adding some static_casts
* removing connection manager usage in client_test to improve readability
* fix nits
* fix minor typo
* fix sanity checks
* update fuzzing_event_engine_test to new api
* update event engine time type
Co-authored-by: Vignesh2208 <Vignesh2208@users.noreply.github.com>
This change includes:
* adding a cert file path for MacOS
* updating related test to run on MacOS too
* s/linux/supported/ since this now includes more platforms
* regenerating files affected by the name change
* [resource_quota] Periodic update tracker
Periodic update type that tries to do updates at some measurable
timescale without polling the current time every tick, instead trying to
approximate how many ticks will be required to fill the desired
duration.
The intent here is that we use this in RQ to donate memory back
periodically whilst keeping timer checks off the fast path.
* fix
* Automated change: Fix sanity tests
* review feedback
* fix
* fix
* review feedback
* Automated change: Fix sanity tests
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* implement UniqueTypeName API
* convert security code to use UniqueTypeName
* change subchannel data producer API to use UniqueTypeName
* sanitize
* add missing build dep
* fix credentials_test
* fix certificate_provider_store_test
* fix tls_security_connector_test
* attempt to fix windows build
* avoid unnecessary allocation
* work around MSVC 2017 bug
* sanity
* change factory to not be templated
* fix sanity
* fix bug in chttp2 connector that used server creds instead of channel creds
* add missing build dep
* simplify API
* Initial skeleton for outlier detection
* fixing code review comments (modifying child policy)
* Skeleton and all tests passing except for 1
* small code review comments fix
* Adding the parsing of the policy in cds and put it in discovery
mechansim json format
* Parsing outlier detection json policy from parent
* Adding parsing of the updates
* Added Subchannel wrapper and watcher wrapper: and all states pass
through and all tests still pass
* added framework to do eject and uneject
* fixing code review comments
* restore a test
* fixing code review comments
* taking care of code review comments
* removing debug code and rebuild build files
* fixing according to code review comments
* fixing code review comments
* Adding address to subchannel map
* addressing code review comments
* adding call counter
* Refcount SubchannelState (in the map) and store them in Subcahnnel Wrapper
* fixing counterss
* Call counter and tracker skleton added
* Call counter
* addressing code review comments
* addressing code review comments
* Added CallCounter and timer
* fixing sanity; but more importantly: taking out timer temporarly as it
was causing test failures.
* sanity
* fixing according to code review comments
* addressing code review comments
* all algorithms implemented
* addressing code review comment about starting the timer
* protect private vars
* small fix
* Added one more corner case
* fixing EjectionTimer
* Fixing according to code review suggestions.
* fixing according to code reveiw comments
* taking care of code review comments
* fixing sanity issues
* Adding proto to tests
* First test
* Fixing according to code review comments
* Tests all working now
* fixing a crash
* fixing build files
* fixing sanity
* sanity
* Simplifying tests
* merge and update
* format
* sanity and format
* Fixing asan error
* fixing parsing logic and error handling
* 6 more tests done
* Added verifying unejection to tests
* Added all the tests
* fixing according to code review comments
* fixing asan and ubsan
* Fixing tests according to code review comments
* Added both algorithm tests
* added percentage enforcement tests
* fixing tsan error
* keeping debug, but fix warning
* remove debugs
* fixing IWYU and build errors after
* test comments change only but very important
* fixing code review comments
* one more refactorying of util function
* Removed debugs and added one more helper method
* one more logic fix
* Fixing last bit of code review comments and added disable tests
* fixing code review comments
* fixing IWYU
* sanity format
* protecting the feature with environment var:
registering policy and generating policy
* added a todo according to code review comments
* fixing a clang finding at import time
* build fix after synching to latest
* move some code around
* remove num_backends parameter from XdsEnd2endTest
* remove use_xds_enabled_server param from XdsEnd2endTest
* remove xds_resource_does_not_exist_timeout_ms param from XdsEnd2endTest
* remove client_load_reporting_interval_seconds param from XdsEnd2endTest
* start moving CreateAndStartBackends() into individual tests
* finish moving CreateAndStartBackends() into individual tests
* remove unused variable
* remove SetEdsResourceWithDelay
* fix test flake
* clang-tidy
* clang-format
* move test framework to its own library
* fix build
* clang-format
* fix windows build
* rename TestType to XdsTestType
* move BackendServiceImpl inside of BackendServerThread
* clang-format
* move AdminServerThread to CSDS test suite
* move RLS tests to their own file
* remove unnecessary deps
* generate_projects
* move some code around
* remove num_backends parameter from XdsEnd2endTest
* remove use_xds_enabled_server param from XdsEnd2endTest
* remove xds_resource_does_not_exist_timeout_ms param from XdsEnd2endTest
* remove client_load_reporting_interval_seconds param from XdsEnd2endTest
* start moving CreateAndStartBackends() into individual tests
* finish moving CreateAndStartBackends() into individual tests
* remove unused variable
* remove SetEdsResourceWithDelay
* fix test flake
* clang-tidy
* clang-format
* move test framework to its own library
* fix build
* clang-format
* fix windows build
* move fault injection tests to their own file
* rename TestType to XdsTestType
* move BackendServiceImpl inside of BackendServerThread
* clang-format
* generate_projects
* appease clang-tidy
* move AdminServerThread to CSDS test suite
* remove unnecessary deps
* generate_projects
* don't mark test as flaky
* Revert "Revert "ORCA: implement ORCA RPC service for OOB backend metric reporting (#29215)" (#29351)"
This reverts commit 71b355624f.
* move ORCA service to its own BUILD rule
* ORCA: implement ORCA RPC service for OOB backend metric reporting
* fix unused parameter error
* gen_upb_api
* add missing build deps
* increase test timing fudge factor
* add missing copyright header
* buildifier
* don't register as a generic service
* report interval defaults to min interval
* don't regenerate the response proto unless something changed
* use INTERNAL for proto parsing failure
* use absl::Duration in public API
* Refactor end2end tests to exercise each EventEngine
* fix incorrect bazel_only exclusions
* Automated change: Fix sanity tests
* microbenchmark fix
* sanitize, fix iOS flub
* Automated change: Fix sanity tests
* iOS fix
* reviewer feedback
* first pass at excluding EventEngine test expansion
Also caught a few cases where we should not test pollers, but should
test all engines. And two cases where we likely shouldn't be testing
either product.
* end2end fuzzers to be fuzzed differently via EventEngine.
* sanitize
* reviewer feedback
* remove misleading comment
* reviewer feedback: comments
* EE test_init needs to play with our build system
* fix golden file test runner
Co-authored-by: drfloob <drfloob@users.noreply.github.com>
* Revert "Revert "HTTP2: Add graceful goaway (#29050)" (#29187)"
This reverts commit 882f64e376.
* Make sure fds are added to pollsets before the main test contents
* Call finalization for promises
* comment
* split out and test
* dont use promise_detail:: directly
* fix
* Automated change: Fix sanity tests
* fix
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* Move XdsChannelCreds to CoreConfiguration
* move xDS channel creds files to src/core/lib/security/credentials/xds
* Change back to returning a RefCountedPtr.
* make remove "xds_" from xds_channel_* files.
* Renamed to address comments.
* clang fix
* Fix another clang error
* HTTP2: Add GrpcNetworkStreamState metadata for calls that are not sent on wire and for those that are not seen by server
* Generate projects
* clang-tidy
* Fix test
* clang-tidy
* Add a negative test
* Fix for windows
This avoids having to do a cherry-pick import, and is harmless since
there are no dependencies yet on the EventEngine. This test will be
re-enabled shortly after both the import and related changes are
finished.
* Adding TLS Key export logic to core and c++ wrappers
* Adding and end2end cpp tls key export test and updating broken test due to interface changes
* regenerate projects
* updating tls key export core logic with addition of APIs to grpc_security.h
* undoing changes to tls_security_connector_test
* regenerate projects
* changing the logging format enum name as per GRFC comments
* regenerate projects
* removing some commented code
* updating changes as per review comments
* adding GRPCAPI annotations to functions defined in grpc_security.h
* regenerate projects
* fixed some code styling issues
* removing grpc_security.h include from tls_credentials_options.h
* updating files as per review comments
* minor fixes
* moving some code around
* removing key log format from tls session key log config and converting it to a simple string
* regenerate projects
* fixing mistakes in recent merge with master
* regenerate projects
* regenerate projects
* fixing some distrib and snity errors
* fixing formatting errors
* fixing more sanity checks and raising supported openssl versions to 1.1.1
* updating min supported openssl version to 1.1.1
* updating min supported openssl version in tls_key_export_test
* updating test to fix incorrect vector initialization
* updating as per latest comments
* fixing sanity checks
* addressing review comments
* fixing sanity checks
* fixed c++ comment style
* Automated change: Fix sanity tests
* fixing review comments
Co-authored-by: Vignesh2208 <Vignesh2208@users.noreply.github.com>
Eliminate slice interning, and structures in slices to support it.
Reduces grpc_slice_refcount from 40 bytes (+ a required 8 bytes elsewhere) to 16 bytes.
Removes a pointer dereference for every slice ref/unref.
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
This patch introduces a factory to allow supporting custom xDS channel
creds. Three types currently supported (fake, insecure, google_default)
are registered by default for backward-compatibility.
* Revert "Revert "user-agent metadata trait, also: grpc_core::Slice is born (#27770)" (#28108)"
This reverts commit 89d08dad9d.
* will it blend
* will it blend
* will it blend
* lcnagfmt
* sanity
* will it blend
* sleep @ end
* will it blend
* Revert "sleep @ end"
This reverts commit d6bca8ed3d.
* review feedback
* review feedback
* Check if memory owner available prior to polling it
The transport may drop the memory owner during its destruction sequence
* tcp_fix
* Revert "Revert "New resource quota integration (#27643)" (#28014)"
This reverts commit 0ea2c37263.
* clang-format
* fix-path
* fix
* Reintroduce the EventEngine default factory
An application can provide an EventEngine factory function that allows
gRPC internals to create EventEngines as needed. This factory would be
used when no EventEngine is provided for some given channel or server,
and where an EventEngine otherwise could not be provided by the
application. Note that there currently is no API to provide an
EventEngine per channel or per server.
I've also deleted some previous iterations on global EventEngine and
EventEngine factory ideas. This new code lives in a public API, and
coexists with iomgr instead of being isolated to an EventEngine-specific
iomgr implementation.
* add proper namespaces, and fix description
* put factory functions in their own file (for replaceability)
* add synchronization
* generate_projects.sh
* extract event_engine_base and event_engine_factory targets
Also separate iomgr/event_engine files in the BUILD, with comments
* gpr_platform
* move all EE factory declarations to event_engine_base
Makes internal hackery easier.
* add missing deps
* reorder dep alphabetically
* comment style change
This commit is part of the effort to create binder channel with
GRPC_CLIENT_CHANNEL type.
The resolver will be used during name resolution, and the result will
later be used to identify the corresponding endpoint binder in
SubchannelConnector.
Besides the unit test, this change is tested with other changes locally
end to end on real device.
The pool serves as a buffer for interaction between C++ and Java.
The buffer can let us avoid calling into Java code in channel connector
implementation (which has not been merged yet), simplifies interaction
between C++ and Java.
Temporary changes are made to channel_create.cc to keep the example apps
working but they will be rewrite after we start creating client channel
instead of direct channel.
* RLS LB policy plugin implementation
* Add RLS tests
* rls proto
* generate projects
* Fix adding child policy to child policy map
* Fix issues
* Fix rotten bits
* generate projects
* revert some strange changes
* First batch of fixes
* second batch of addressing comments
* generate_projects
* 3rd batch of fixes
* generate_projects
* 4th batch of fix
* 5th batch of fix
* Empty commit to trigger github PR refresh
* Add insecure build
* generate project
* update picker in a closure
* mutex guard audit
* clang-format
* Aggregate child policy states
* Use OrphanablePtr on ChildPolicyOwner()
* More fixes on comments
* Remove include grpc_security from rls.cc
* Fix key builder test as we removed RlsFindPathFromMetadata
* Update rls proto and add multi-target capability
* code changes to fix build after merging master
* remove support for insecure builds
* WIP
* add LB policy metadata lookup API
* add API for creating errors from C++ strings
* add missing include
* finished updating JSON parsing
* use DualRefCounted<> for ChildPolicyWrapper
* use grpc_core::Mutex instead of std::recursive_mutex
* add lock annotations and fix some lifetime issues
* misc fixes and cleanups
* simplify child policy connectivity state machine
* add comment about hopping into ExecCtx
* don't use wait_for_ready for RLS call
* extraKeys and constantKeys support
* add lock annotations for child policy wrapper connectivity state
* improve logging
* simplify child policy state handling in picker
* use C++ style comments
* remove rls_config.proto, since it's not needed
* rewrote tests
* fix build
* fix copyright headers
* clang-format
* appease clang-tidy
* remove unnecessary dependendency on grpclb
* buildifier
* remove illegal term
* remove unneeded includes
* clang-format
* fix clang-tidy
* fix build
* move class declaration into .cc file
* clang-tidy again
* fix build
* reorganize code and misc cleanups
* clang-format
* fixed a bunch of asan bugs; still some left to debug
* fix asan problem
* make test work on IPv6-only machines
* move LRU list handling into Cache::Entry and add lock annotations
* use preincrement instead of postincrement
* fix deadlock
* add more FIXMEs
* clean up backoff timer logic
* fix build from merge
* clang-format
* fix include path to work on podspec builds
* clean up picker logic
* clang-format
* update rls.proto
* populate reason and stale_header_data in RLS request
* improve logging and fix some cache size type issues
* clang-format
* add some TODOs about a better way to expose channel creds to LB policies
* centralize handling of channel's default authority
* clang-format
* fix backup poller bug
* fix handling of call creds and authority on RLS channel
* remove unused params
* clang-format
* use two-phase update for child policies to avoid deadlocks
* fix portability issue
* minor cleanups
* update for change in grpc_error_get_status() API
* change test to store RLS requests and responses in proto form
* clang-format
* account for test slowdown factor in timeouts
* fix tsan failure and channelz linkage
* fix RLS authority death test
* fix test to not try to connect to target that should not work
* remove illegal term
* cacheSizeBytes is a required field
* add missing BUILD deps from merge
* add tests for connectivity state reporting and fix bugs found
* fix BUILD sanity
* buildifier
* fix BUILD package path
* fix bugs related to child policy lifetime and updates
* remove unnecessary srand() call from grpc_init()
* add test for two cache entries with the same target
* update rls.proto
* change RLS config parser test to support GRPC_ERROR_IS_ABSEIL_STATUS
* update upb codegen
* fix include path
* add env var guard
* avoid duplication of CountedService
* generate_projects
* add upper limit for cacheSizeBytes
* fix build
Co-authored-by: Muxi Yan <mxyan@google.com>
Add `grpc::BinderServerCredentials()` and other related functionalities for the server to listen to binder transactions through a phony "binder port".
The APIs are temporarily placed in internal headers until the corresponding gRFC is merged.
* Avoid copy constructor
```
src/core/lib/gpr/log_linux.cc:78:33: error: copying variable of type 'grpc_core::PthreadTlsImpl<long>' invokes deleted constructor
static GPR_THREAD_LOCAL(long) tid = 0;
^ ~
./src/core/lib/gpr/tls.h:64:3: note: 'PthreadTlsImpl' has been explicitly marked deleted here
PthreadTlsImpl(const PthreadTlsImpl&) = delete;
^
1 error generated.
```
* Generalize pthread TLS to any trivial type
Use multiple pthread keys for types larger than a machine word.
Implement generic timer TLS optimization on all platforms.
This PR imports unit tests and end-to-end tests of binder transport from the internal repository. No further changes will be made to internal repository.
* Add construct/destruct helper functions
Will be used in upcoming promise implementation
* Poll type for promises library
* Library to talk about things that look like promises if you squint
* Helper code for promises to deal with status types generically
* Library to talk about things that make promises
* build
* Join combinator for promise library
* Changes to sync required for promise activities
* sanitized
* Automated change: Fix sanity tests
* Automated change: Fix sanity tests
* Update basic_join.h
* Automated change: Fix sanity tests
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* Poll type for promises library
* Library to talk about things that look like promises if you squint
* Library to talk about things that make promises
* Promises loop construct
* build
* Changes to sync required for promise activities
* sanitized
* add comments
* Automated change: Fix sanity tests
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* Poll type for promises library
* Library to talk about things that look like promises if you squint
* Library to talk about things that make promises
* Promises if construct
* build
* Automated change: Fix sanity tests
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* Poll type for promises library
* Library to talk about things that look like promises if you squint
* Promise helpers, and basic type erasure
* Promise map operator - change return type via a function
* build
* Changes to sync required for promise activities
* sanitized
* Automated change: Fix sanity tests
* fixes
* fixes
* Automated change: Fix sanity tests
* fix build
* review feedback
* Automated change: Fix sanity tests
* comment
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* Poll type for promises library
* Library to talk about things that look like promises if you squint
* Library to talk about things that make promises
* build
* Changes to sync required for promise activities
* sanitized
* remove bad comment
* possible windows fix?
* fix
* ugh
* comment fix
* fix build
* Automated change: Fix sanity tests
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* Poll type for promises library
* Library to talk about things that look like promises if you squint
* Promise helpers, and basic type erasure
* build
* Changes to sync required for promise activities
* sanitized
* Automated change: Fix sanity tests
* suppressions
* try to fix windows failure
* Automated change: Fix sanity tests
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* Limit initial window size increases/decreases by flow control windows of active streams
* Limit stream flow control window updates to maximum allowable
* Reviewer comments
* Alternative way
* Clean-up
* Add tests
* Remove unnecessary cq_verifier
* Generate projects
* Initialize recv_message
* Get around compilation issue
* Test size large
AJ Heller