* Revert "Revert "Add support for systemd socket activation (#30485)" (#31617)"
This reverts commit 867dc6cae2.
* Add checks to unix tests
* Ran generate_projects.sh and fixed styling in test
* Fix variable in unit test
* Use reinterpret_cast in test
* Rebase and fix sanity failures
This is motivated by #31227 which I also encountered recently. I'm not familiar with the call lifecycle details, but it seems clear that there is some kind of race condition between user-initiated cancellations and other completion status updates.
If a user cancels a call before being notified of its completion, I believe it should be considered cancelled unconditionally, regardless of whether the call actually completed successfully before its cancellation status could be set.
So it seems better to re-raise explicitly rather than using self._raise_for_status(), which is a no-op in this race condition and can lead to an UnboundedLocalError.
* rename task-handle mutex
* rename TimerClosure
* tmp commit, building, not tested
* add test for client connection to a non-listening port
* fix posix EE tests
* re-fix windows test suite after posix compatibility
* (unfinished, backup): passing the suite's NonExistingListener client test
* remove redundant windows client test
* integrate IOCP worker loop
* initial commit of echo test tool; fixes
* move echo client to test_suite/tools; I do not yet like the setup, it's about time for a macro that generates all useful test/tool targets
* cleanup
* add --target arg to echo_client. requires URI
* Automated change: Fix sanity tests
* build fixes
* fix
* fix
* reviewer feedback
* warning fix
* delete logic on cancellation
* fix
* cancel connect deadlock; improved template code
* fix build failure with linux environments building windows targets
* fix
* fix
* no ++ for atomics
* remove the test changes, to be landed separately
* Automated change: Fix sanity tests
* remnants
Co-authored-by: drfloob <drfloob@users.noreply.github.com>
* Revert "Revert "xDS stateful session affinity: add config plumbing (#31827)" (#31873)"
This reverts commit 4f15d3dcf9.
* fix build for compilers too dumb to recognize the full set of enum values
* Add type annotation to _channel.py and _server.py
* Undo changes in init.py
* changes based on comments.
* format changes
* fix pytype errors
* Changes based on comments
* Fix circular import
* fix pytype and pylint
* Fix format issues
* Fix some cython type issues
* Change ManagedCallType to IntegratedCallFactory
* fix format
* undo some mistakes
Add a check that SEND_STATUS_FROM_SERVER and RECV_MESSAGE are not in the same batch.
This is necessary pre-work for #31204 and implements part of grpc/proposal#336.
Also eliminates fling instead of updating it:
My expectation is nobody has looked at this corner in many years
It's not a benchmark we want: concentrating on a microbenchmark that doesn't include a binding layer caused us to favor designs that emphasized a lightweight core at the expense of a expensive bindings. We should consider the whole.
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* Actually record CPU usage
* Do the client path too
* Actually build the image from the PR
* Revert "Actually build the image from the PR"
This reverts commit 3afa5cf3c3.
* Support importing on Windows
* Declarative JSON parser
* Automated change: Fix sanity tests
* fix
* shrinking stuff a little
* static vtables
* separate fns
* simpler?
* make maps work
* windows fixes
* Automated change: Fix sanity tests
* simplify code
* Automated change: Fix sanity tests
* vtable-test
* dont always create vec/map impls for every type
* comments
* make error consistent
* move method private
* progress
* durations!
* Automated change: Fix sanity tests
* fix
* fix
* fix
* Automated change: Fix sanity tests
* post-load
* Automated change: Fix sanity tests
* document JsonPostLoad() and add static_assert
* don't copy field names, to avoid length limitations
* use absl::Status
* accept either string or number for numeric values
* add test for direct data member of another struct type
* remove unused method
* add support for retaining part of the JSON wirthout processing
* update test for changes in Json::Parse() API
* add absl::optional support
* Automated change: Fix sanity tests
* fix tests, improve error messages, and add overload to parse to existing object
* remove overload of LoadFromJson()
* change special case for Json to instead use Json::Object
* rename resolver_result_parsing to client_channel_service_config
* split up service_config_test into a separate test for each component
* convert client channel service config parser to new API
* fix build
* converted retry global params
* convert retry method-level parsing, but still need to find a way to control parsing via a channel arg
* improve error structure, add missing types, and improve tests
* clang-format
* Automated change: Fix sanity tests
* fix build
* add LoadJsonObjectField(), add LoadFromJson() overload that takes an ErrorList parameter, and add tests for parsing bare top-level types
* fix msan
* Automated change: Fix sanity tests
* fix error message
* Automated change: Fix sanity tests
* fix test
* add ability to disable fields
* plumb in channel args to disable parsing
* Automated change: Fix sanity tests
* use const char* instead of absl::string_view for enable_key
* fix resolver_component_test
* Automated change: Fix sanity tests
* work around mac build problem
* Automated change: Fix sanity tests
* work around gcc6 problem
* Automated change: Fix sanity tests
* fix build
* fix build
* don't use alternative builder in tests
* convert message size service config parser
* convert fault injection service config parser
* rename files
* add specialization for unique_ptr
* avoid moves in client channel service config parser
* avoid moves in retry service config parser, and do some cleanup
* avoid moves in message_size service config parser
* avoid moves for fault injection service config parser, and use internal channel arg
* convert rbac service config parser
* clang-format
* WIP
* convert top-level service config parser
* clang-format
* fix build
* fix rbac service config parser test
* fix signed-ness problem and reversed-conditional bug
* fix unused param
* fix json_string method
* fix max message length defaults
* add copy ctors to appease windows compiler
* fix RLS LB config parser test
* fix name resolution test
* fix build
* work around gmock portability bug
* fix sanity
* add missing build dep
* make RBAC principal and permissions movable, not copyable
* Revert "make RBAC principal and permissions movable, not copyable"
This reverts commit 53315bccc9.
* attempt to simplify HeaderMatcher and StringMatcher parsing
* more bloat reduction in RBAC service config parser
* fix sanity
* add missing build dep
* attempt work-around for MSVC bug
* Revert "attempt work-around for MSVC bug"
This reverts commit e54c89e1e4.
* attempt work-around for Windows build problem
* try another work-around
* fix sanity
* appease clang-tidy
* generate_projects
* attempt to fix Windows build
* more windows fixes
* more windows fix
* yet more windows fixes
* try without noexcept
* remove unnecessary boilerplate
* code review changes
* fix breakage
Co-authored-by: Craig Tiller <craig.tiller@gmail.com>
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
Co-authored-by: Craig Tiller <ctiller@google.com>
Co-authored-by: markdroth <markdroth@users.noreply.github.com>
* XdsBootstrap: move two more methods out of the interface
* Automated change: Fix sanity tests
* XdsClient: add unit test
* Automated change: Fix sanity tests
* fix memory leaks
* add helper method
* add unsubscription
* add test for multiple subscriptions
* clang-format
* fix build
* fix flakiness
* add checking for other node fields
* add v2 test
* add response builder
* add test for update from server
* add test for update containing only changed resources
* clang-format
* fix build
* add test for resource not existing upon subscription
* add test for stream closed by server
* add test for multiple watchers for the same resource
* add test for connection failure
* clang-format
* add test for resources wrapped in Resource wrapper message
* add test for resource validation failure
* add test for multiple invalid resources, and fix a case in XdsClient
* add test for validation failure for already-cached resource
* add test for server not resending resources after stream disconnect
* clang-format
* fix XdsClient to report channel errors to newly started watchers
* fix XdsClient to send cached errors/does-not-exists to newly started watchers
* fix watcher to ensure events arrive in the expected order
* fix tests
* clang-format
* add test for multiple resource types
* fix xds_cluster_e2e_test
* Automated change: Fix sanity tests
* cleanup
* add federation tests
* clang-format
* remove now-unnecessary XdsCertificateProviderPluginMapInterface
* code review comments
* simplify XdsResourceType::Decode() API
* XdsClient: add unit tests for XdsClusterResourceType
* add XdsClient with gRPC bootstrap config
* add LB policy tests
* started adding CertificateProvider tests
* update for recent API changes
* fix merge bugs
* xDS resource validation: identify extensions by type_url instead of name
* fix build
* migrate to ValidationErrors
* add xds_common_types_test
* finish TLS tests and add LRS tests
* move ScopedExperimentalEnvVar to its own library and remove redundant e2e tests
* add circuit breaking and outlier detection tests
* add validation to outlier detection LB policy parsing
* clang-format
* Automated change: Fix sanity tests
* fix signedness
* fix sanity
* xDS: implement xds_wrr_locality LB policy and return xDS LB config from XdsClient
* fix unused parameter
* fix sanity
* fix test
* Automated change: Fix sanity tests
* fix aggregate cluster bug
* Automated change: Fix sanity tests
* absl::make_unique -> std::make_unique
* fix sanity
* fix sanity
* iwyu
* iwyu
* update code for XdsResourceTypeImpl changes
Co-authored-by: markdroth <markdroth@users.noreply.github.com>
* begin c++
* Automated change: Fix sanity tests
* progress
* progress
* missing-files
* Automated change: Fix sanity tests
* moved-from-stats
* remove old benchmark cruft, get tests compiling
* iwyu
* Automated change: Fix sanity tests
* fix
* fix
* fixes
* fixes
* add needed constructor
* Automated change: Fix sanity tests
* iwyu
* fix
* fix?
* fix
* fix
* Remove ResetDefaultEventEngine
Now that it is a weak_ptr, there's no need to explicitly reset it. When
the tracked shared_ptr is deleted, the weak_ptr will fail to lock, and a
new default EventEngine will be created.
* forget existing engine with FactoryReset
* add visibility
* fix
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
Co-authored-by: AJ Heller <hork@google.com>
* Re-add compiler protos to grpcio-tools
* Attempt to break
* Actually fail if protobufs don't compile. Who wrote this thing?
* Add an actual test
* yapf
* Whoops. Thought I'd reverted you
* Please the formatter gods
* Support Python 3.11
* Update build images for 3.11
* Whoopsie
* The architecture of this thing is garbage
* Silence ownership warning
* Account for change in git behavior
* Fix directory
* I am in great pain
* Update Windows and arm linux
* Agh
* Clean up
* [cleanup] Remove profiling timers
- nobody has used this system in years
- if we needed it, we'd probably rewrite it at this point to be something more modern
- let's remove it until that need arises
* fix
* fixes
A (currently) pthread_atfork-based fork support mechanism, allowing EventEngines - or any other object that wants to implement the Forkable interface - respond to forks.
* Rename the default EventEngine headers
Small cleanup. This code hasn't been related to factories for a month or
two.
* ensure only one target contains default_event_engine.h
* src + hdr in same target
* include guards
* Update third_party/protobuf
* run tools/distrib/python/make_grpcio_tools.py
* regenerate protos for ruby, php
* update build_handwritten.yaml
* regenerate projects
* Build - Use :well_known_type_protos instead of :well_known_protos
* Fix target
* Update upb
* Update Python for Protobuf 4.21 (#140)
* Update protobuf dependency on grpcio-tools
* Off by one
* Drop python 3.6 support
* Try upgrading pip
* And in the other script
* Try to figure out if we're compatible with abi3
* See what we've already got installed
* Update the requirements.txt file I didn't know existed
* And here too
* See what's installed
* Let's try that again
* Remove
* Try to confirm version
* Let me see the generated code
* Fix non-Bazel test runner
* Work for all test directories
* Regenerate example protos
* Clean up
* Generate .pyi files
* Fix type checking and linting
* Exclude pyi files from isort
* Upgrade to 3.21.4
* Update iwyu to get around messy protobuf IWYU rules
Co-authored-by: Richard Belleville <gnossen@gmail.com>
This is a partial fork of the windows iomgr code - specifically the IOCP and Socket pieces - with some improved architecture and encapsulation. And the start of a WindowsEventEngine.
Once this code is used in a gRPC TCP context, I imagine a few issues will shake out. Also, getting sanitizers set up with MSVC will take a bit of work (see a commit referencing abseil and MSVC bugs to hack around).
I forked the IomgrEventEngine's posix poller interfaces in the hope of negotiating compatibility between the platforms, but the interfaces diverged a fair bit, and I'm doubtful we'll be able to use these "pollers" generically in the same TCP code. Reunification might not happen, and that's probably fine, we'll see how similar the TCP code looks once it's fleshed out.
I also extracted the IomgrEventEngine's timer piece into a separate component, usable by both engines.
* xDS: upgrade the commit of the xDS protos that we're using
* Automated change: Fix sanity tests
Co-authored-by: markdroth <markdroth@users.noreply.github.com>
This change includes:
* adding a cert file path for MacOS
* updating related test to run on MacOS too
* s/linux/supported/ since this now includes more platforms
* regenerating files affected by the name change
* [Aio] Ensure Core channel closes when deallocated
* Keep channel object alive
* Let Multicallable objects hold the reference to Channel
* Make YAPF happy
This code is not compiled by default and has fallen out of sync with the
rest of the codebase. There's a good chance it won't be used, given our
current work to use an iomgr-drive EventEngine instead.
This code will continue to live in git history, should we need to bring
pieces of it back.
* Initial skeleton for outlier detection
* fixing code review comments (modifying child policy)
* Skeleton and all tests passing except for 1
* small code review comments fix
* Adding the parsing of the policy in cds and put it in discovery
mechansim json format
* Parsing outlier detection json policy from parent
* Adding parsing of the updates
* Added Subchannel wrapper and watcher wrapper: and all states pass
through and all tests still pass
* added framework to do eject and uneject
* fixing code review comments
* restore a test
* fixing code review comments
* taking care of code review comments
* removing debug code and rebuild build files
* fixing according to code review comments
* fixing code review comments
* Adding address to subchannel map
* addressing code review comments
* adding call counter
* Refcount SubchannelState (in the map) and store them in Subcahnnel Wrapper
* fixing counterss
* Call counter and tracker skleton added
* Call counter
* addressing code review comments
* addressing code review comments
* Added CallCounter and timer
* fixing sanity; but more importantly: taking out timer temporarly as it
was causing test failures.
* sanity
* fixing according to code review comments
* addressing code review comments
* all algorithms implemented
* addressing code review comment about starting the timer
* protect private vars
* small fix
* Added one more corner case
* fixing EjectionTimer
* Fixing according to code review suggestions.
* fixing according to code reveiw comments
* taking care of code review comments
* fixing sanity issues
* Adding proto to tests
* First test
* Fixing according to code review comments
* Tests all working now
* fixing a crash
* fixing build files
* fixing sanity
* sanity
* Simplifying tests
* merge and update
* format
* sanity and format
* Fixing asan error
* fixing parsing logic and error handling
* 6 more tests done
* Added verifying unejection to tests
* Added all the tests
* fixing according to code review comments
* fixing asan and ubsan
* Fixing tests according to code review comments
* Added both algorithm tests
* added percentage enforcement tests
* fixing tsan error
* keeping debug, but fix warning
* remove debugs
* fixing IWYU and build errors after
* test comments change only but very important
* fixing code review comments
* one more refactorying of util function
* Removed debugs and added one more helper method
* one more logic fix
* Fixing last bit of code review comments and added disable tests
* fixing code review comments
* fixing IWYU
* sanity format
* protecting the feature with environment var:
registering policy and generating policy
* added a todo according to code review comments
* fixing a clang finding at import time
* build fix after synching to latest
Rollforward with TCP connect handshaker again(#29111) after fixing broken internal targets.
The changes needed were just visibility changes to the handshaker and the http_connect_handshaker libraries as they are used internally.
* Move unnecessary absl includes from metadata_batch.h
metadata_batch.h is a widely used header, and so any includes it takes
that are for its internal use can negatively impact our build times.
* missing file
* build
* review feedback
Currently the tcp connect is performed in chttp2_connector before the handshaking is triggered. For
use cases where the application wants to perform business logic before
the tcp connection, this is problematic. By moving the TCP connect into
its own handshaker and registering it by default at the beginning, this
allows applications to add a new handshaker at the beginning allowing
handshaker logic before a TCP connect.
This approach has the advantage of slightly simplifying the logic in
tcp_connect_handshaker and httpcli as tcp_connect/callback can be
removed.
As the TCP connect needs parameters like resolved_addr,
interested_parties, a new struct called connection args is created as a
member for Handshaker Args.
For server handshakers most of the arguments here are not directly
useful, other than the deadline.
* WIP: add OOB backend metric API for LB policies
* fix some includes
* minor fixes
* picking this up again...
* more WIP
* health checking: cancel stream if response message fails to parse
* basic structure in place, but still have synchronization issues to address
* ORCA: implement ORCA RPC service for OOB backend metric reporting
* fix unused parameter error
* gen_upb_api
* add missing build deps
* increase test timing fudge factor
* add missing copyright header
* fix build and locking problems
* clang-format
* document API
* buildifier
* add test, but doesn't build yet
* new test working, but broke existing test, and need to fix server API
* don't register as a generic service
* update test for new orca service registration API
* fix build
* sanitize
* report interval defaults to min interval
* add channel trace event on UNIMPLEMENTED
* don't regenerate the response proto unless something changed
* add missing build dep
* fix comment
* Initial unit testbench
* Tentative fix
* Reduce scope of fix. Clean up test
* Formatting
* Protect writing to RPC status store
* Don't start client until server is running
* Typo
* Remove redundant log
* More formatting
* Review comments
* [PSM interp] Set hostname in initial metdata for Python servers
* DO-NOT-MERGE: hijact psm-security-python for one test case
* Revert "DO-NOT-MERGE: hijact psm-security-python for one test case"
This reverts commit 17b8af9ab9.
* Metadata is a tuple of tuple not a tuple
* Initial structure for RLS
* Adding and building the proto to parse the Any proto for the plugins
* re-org
* Parsing the plugin
* Parsing more into json
* Parsed proto to json
* small cleanup
* Adding prefix
* Added new rls_experimental policy
* build files
* Fixing according to code review comments
* code review comments
* Adding sym changes
* adding action name check
* fixing code review comments.
* fixing unused var error
* clean up
* fixing code review comments
* fixing code review comments
* fixing according to code review comments.
* Remove unnecessary include
* small fix
* generate more, hard-code less
* Moving to using absl::variant
* absl::string_view and absl::variant of vector of std::string are not
playing nice together.
* fixed variant
* Using absl::variant now
* Checkint used plugins
* Refactor Parsing code and separating out Parsing of the plugin
* Fixing code review comments
* code review comments
* fixing code review comments.
* Addressing code review comments
* First end-to-end test
* generated build files
* commit generated files via tools/codegen/core/gen_upb_api.sh
* Fixing rls policy parsing tests
* Restore checks for the test server
* Refactor rls_server
* added keys to rls request
* fixing small logic error
* Complete the test using all the keys
* Separating out RLS test and rls_server thread
* sanity errors
* generated build files
* Complete the rest of the tests and sanity cleanup
* fixing code review comments: using upb_JsonEncode now!
* fixing code review comments
* fixing code review comments
* Fixing code review comments
* misisng fix
* simplifying tests
* simplify tests 2
* Linking in the correct proto for rls_config
* restore metadata check
* Add disable test
* Fixing RLS test and removing environment var that is no longer necessary
* Fixing "Wrong type" type of tests after json parsing change to accept
STRING for number
* adding json_encode.h/c to src/upb/gen_build_yaml.py and generate
necessary files.
* Fixing un-used var error
* fixing sanity errors
* Fixing the upb encoding buffer
* Fixing code review comments.
* Adding nack test for unkonwn plugin proto
* Last bit of code review comments
* fixing unused variable
* WIP
* Add gevent test suite run under Bazel.
* Fix things up
* Yapf
* Fix up Bazel files
* Make py_grpc_test fancier
* Attempt to fix Windows RBE
* Attempt to kick GitHub
* Fix Python 2 runs
* Yet more fixes
* And the patch file too
* I am an idiot
* Mark gevent tests flaky
* Try to make rules_python more tolerant
* Typo
* Exclude reconnect test from gevent
* Remove unnecessary parts of patch
* Buildifier
* You saw nothing
* isort
* Move py_grpc_test to an internal-only file
* Review comments
* More reviewer comments
* Review
* Add initial changes for gevent
* WIP. Run completion_queue_next in a threadpool
* WIP.
* WIP
* Re-remove skip annotation
* Finally working
* Reactivate tests
* Clean up
* Move C++ threading utilities to grpc.pxi
* Unbreak sync stack
* Refix test flake
* WIP. Trying to get things working properly
* Move test stuff to test runner
* Clean up
* Can't handle exceptions if you don't compile with exceptions
* Remove debug stuff unintentionally left in
* Add greenlet switch loggging and fix threading issue
* Only run a greenlet scheduling greenlet when there are open channels
* Format
* Add threadpool modifications to old runner
* And actually import gevent
Christopher S