* Initial structure for RLS
* Adding and building the proto to parse the Any proto for the plugins
* re-org
* Parsing the plugin
* Parsing more into json
* Parsed proto to json
* small cleanup
* Adding prefix
* Added new rls_experimental policy
* build files
* Fixing according to code review comments
* code review comments
* Adding sym changes
* adding action name check
* fixing code review comments.
* fixing unused var error
* clean up
* fixing code review comments
* fixing code review comments
* fixing according to code review comments.
* Remove unnecessary include
* small fix
* generate more, hard-code less
* Moving to using absl::variant
* absl::string_view and absl::variant of vector of std::string are not
playing nice together.
* fixed variant
* Using absl::variant now
* Checkint used plugins
* Refactor Parsing code and separating out Parsing of the plugin
* Fixing code review comments
* code review comments
* fixing code review comments.
* Addressing code review comments
* First end-to-end test
* generated build files
* commit generated files via tools/codegen/core/gen_upb_api.sh
* Fixing rls policy parsing tests
* Restore checks for the test server
* Refactor rls_server
* added keys to rls request
* fixing small logic error
* Complete the test using all the keys
* Separating out RLS test and rls_server thread
* sanity errors
* generated build files
* Complete the rest of the tests and sanity cleanup
* fixing code review comments: using upb_JsonEncode now!
* fixing code review comments
* fixing code review comments
* Fixing code review comments
* misisng fix
* simplifying tests
* simplify tests 2
* Linking in the correct proto for rls_config
* restore metadata check
* Add disable test
* Fixing RLS test and removing environment var that is no longer necessary
* Fixing "Wrong type" type of tests after json parsing change to accept
STRING for number
* adding json_encode.h/c to src/upb/gen_build_yaml.py and generate
necessary files.
* Fixing un-used var error
* fixing sanity errors
* Fixing the upb encoding buffer
* Fixing code review comments.
* Adding nack test for unkonwn plugin proto
* Last bit of code review comments
* fixing unused variable
* WIP
* Add gevent test suite run under Bazel.
* Fix things up
* Yapf
* Fix up Bazel files
* Make py_grpc_test fancier
* Attempt to fix Windows RBE
* Attempt to kick GitHub
* Fix Python 2 runs
* Yet more fixes
* And the patch file too
* I am an idiot
* Mark gevent tests flaky
* Try to make rules_python more tolerant
* Typo
* Exclude reconnect test from gevent
* Remove unnecessary parts of patch
* Buildifier
* You saw nothing
* isort
* Move py_grpc_test to an internal-only file
* Review comments
* More reviewer comments
* Review
* Add initial changes for gevent
* WIP. Run completion_queue_next in a threadpool
* WIP.
* WIP
* Re-remove skip annotation
* Finally working
* Reactivate tests
* Clean up
* Move C++ threading utilities to grpc.pxi
* Unbreak sync stack
* Refix test flake
* WIP. Trying to get things working properly
* Move test stuff to test runner
* Clean up
* Can't handle exceptions if you don't compile with exceptions
* Remove debug stuff unintentionally left in
* Add greenlet switch loggging and fix threading issue
* Only run a greenlet scheduling greenlet when there are open channels
* Format
* Add threadpool modifications to old runner
* And actually import gevent
* Move XdsChannelCreds to CoreConfiguration
* move xDS channel creds files to src/core/lib/security/credentials/xds
* Change back to returning a RefCountedPtr.
* make remove "xds_" from xds_channel_* files.
* Renamed to address comments.
* clang fix
* Fix another clang error
Partially collapse `event_engine_factory.cc` into `event_engine.cc`. Add a
new function `DefaultEventEngineFactory` which is used to set a default
event engine factory at link time, separate from the factory that can be
set at run time. Implemenet this function in
`default_event_engine_factory.cc`.
This allows alternative default event engine factories to be implemented
without requiring the duplication of the implementations of
`SetDefaultEventEngineFactory`, `CreateEventEngine`, and
`GetDefaultEventEngine`.
* Adding TLS Key export logic to core and c++ wrappers
* Adding and end2end cpp tls key export test and updating broken test due to interface changes
* regenerate projects
* updating tls key export core logic with addition of APIs to grpc_security.h
* undoing changes to tls_security_connector_test
* regenerate projects
* changing the logging format enum name as per GRFC comments
* regenerate projects
* removing some commented code
* updating changes as per review comments
* adding GRPCAPI annotations to functions defined in grpc_security.h
* regenerate projects
* fixed some code styling issues
* removing grpc_security.h include from tls_credentials_options.h
* updating files as per review comments
* minor fixes
* moving some code around
* removing key log format from tls session key log config and converting it to a simple string
* regenerate projects
* fixing mistakes in recent merge with master
* regenerate projects
* regenerate projects
* fixing some distrib and snity errors
* fixing formatting errors
* fixing more sanity checks and raising supported openssl versions to 1.1.1
* updating min supported openssl version to 1.1.1
* updating min supported openssl version in tls_key_export_test
* updating test to fix incorrect vector initialization
* updating as per latest comments
* fixing sanity checks
* addressing review comments
* fixing sanity checks
* fixed c++ comment style
* Automated change: Fix sanity tests
* fixing review comments
Co-authored-by: Vignesh2208 <Vignesh2208@users.noreply.github.com>
Eliminate slice interning, and structures in slices to support it.
Reduces grpc_slice_refcount from 40 bytes (+ a required 8 bytes elsewhere) to 16 bytes.
Removes a pointer dereference for every slice ref/unref.
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
This patch introduces a factory to allow supporting custom xDS channel
creds. Three types currently supported (fake, insecure, google_default)
are registered by default for backward-compatibility.
* Fix the distutil install issue
* Try to move setuptools install forward
* Try pin the distribution tools
* Enforce local distutil
* Fix script order of definition
* Add comment
* Try to fix the setuptool monkey patch
* Clean-up imports for commands.py
* Pin 59.6.0 instead of 59.7.0 (incomplete release)
* WIP
* introduce XdsResourceType API and change Listener parsing to use it
* converted RouteConfig parsing
* convert cluster and endpoint parsing
* cleanup
* clang-format
* attempt to work around compiler problems
* move XdsResourceType to its own file, and move endpoint code out of XdsApi
* move cluster parsing to its own file
* move route config parsing to its own file
* move listener parsing to its own file
* clang-format
* minor cleanup
* plumbed XdsResourceType throughout XdsClient
* a bit of cleanup
* more cleanup
* construct full resource names before calling XdsApi::CreateAdsRequest()
* remove some unneeded code
* clean up includes and have XdsResourceType initialize the upb symtab
* more cleanup of unnecessary code
* more cleanup
* update comment
* clang-format
* add missing virtual dtor
* fix build
* remove comment
* add missing virtual dtor
* introduce XdsResourceType API and change Listener parsing to use it
* converted RouteConfig parsing
* convert cluster and endpoint parsing
* cleanup
* clang-format
* attempt to work around compiler problems
* move XdsResourceType to its own file, and move endpoint code out of XdsApi
* move cluster parsing to its own file
* move route config parsing to its own file
* move listener parsing to its own file
* clang-format
* minor cleanup
* remove comment
* add missing virtual dtor
* Fix status code for resource exhaustion
* Revert "Revert "Move a bunch of slice typed metadata to new system (#28107)" (#28208)"
This reverts commit 7717587063.
* fix test
* Check if memory owner available prior to polling it
The transport may drop the memory owner during its destruction sequence
* tcp_fix
* Revert "Revert "New resource quota integration (#27643)" (#28014)"
This reverts commit 0ea2c37263.
* clang-format
* fix-path
* fix
* Rename the source files for ChannelArgsEndpointConfig
Previous naming was non-descript
* generate_projects
* add missing file to BUILD, and generate_projects.sh
* correct include guards
* Reintroduce the EventEngine default factory
An application can provide an EventEngine factory function that allows
gRPC internals to create EventEngines as needed. This factory would be
used when no EventEngine is provided for some given channel or server,
and where an EventEngine otherwise could not be provided by the
application. Note that there currently is no API to provide an
EventEngine per channel or per server.
I've also deleted some previous iterations on global EventEngine and
EventEngine factory ideas. This new code lives in a public API, and
coexists with iomgr instead of being isolated to an EventEngine-specific
iomgr implementation.
* add proper namespaces, and fix description
* put factory functions in their own file (for replaceability)
* add synchronization
* generate_projects.sh
* extract event_engine_base and event_engine_factory targets
Also separate iomgr/event_engine files in the BUILD, with comments
* gpr_platform
* move all EE factory declarations to event_engine_base
Makes internal hackery easier.
* add missing deps
* reorder dep alphabetically
* comment style change
* Fix all lint errors in repo.
* Use strict buildifier by default
* Whoops. That file does not exist
* Attempt fix to buildifier invocation
* Add missing copyright
* ChannelStackModifier class
* Regenerate projects
* clang-tidy
* Use CoreConfiguration for inserting xDS HTTP filters
* New lines
* Move test to test/core/xds
* Add comment for placement of filter stack
* Fix sanity
* Fix memory leak - destroy builder
* Don't build xds_channel_stack_modifier_test on non-bazel build systems due to census dependency
This commit is part of the effort to create binder channel with
GRPC_CLIENT_CHANNEL type.
The resolver will be used during name resolution, and the result will
later be used to identify the corresponding endpoint binder in
SubchannelConnector.
Besides the unit test, this change is tested with other changes locally
end to end on real device.
* Don't leave pending tasks on the asyncio queue
The results of these pending tasks are not needed, leaving them
on the queue grows the size of the queue until the call completes.
This fix slows the growth of the memory in the test example.
* Address 'leaking' Futures from cygrpc
Cancelling unneeded Tasks is not sufficient as this leaves behind
cancelled Futures in the cygrpc layer, which still occupy memory.
Instead, avoid creating unneeded tasks in the first place.
* Address review comments
1. Ignore unused return values
2. Fix formatting
* RLS LB policy plugin implementation
* Add RLS tests
* rls proto
* generate projects
* Fix adding child policy to child policy map
* Fix issues
* Fix rotten bits
* generate projects
* revert some strange changes
* First batch of fixes
* second batch of addressing comments
* generate_projects
* 3rd batch of fixes
* generate_projects
* 4th batch of fix
* 5th batch of fix
* Empty commit to trigger github PR refresh
* Add insecure build
* generate project
* update picker in a closure
* mutex guard audit
* clang-format
* Aggregate child policy states
* Use OrphanablePtr on ChildPolicyOwner()
* More fixes on comments
* Remove include grpc_security from rls.cc
* Fix key builder test as we removed RlsFindPathFromMetadata
* Update rls proto and add multi-target capability
* code changes to fix build after merging master
* remove support for insecure builds
* WIP
* add LB policy metadata lookup API
* add API for creating errors from C++ strings
* add missing include
* finished updating JSON parsing
* use DualRefCounted<> for ChildPolicyWrapper
* use grpc_core::Mutex instead of std::recursive_mutex
* add lock annotations and fix some lifetime issues
* misc fixes and cleanups
* simplify child policy connectivity state machine
* add comment about hopping into ExecCtx
* don't use wait_for_ready for RLS call
* extraKeys and constantKeys support
* add lock annotations for child policy wrapper connectivity state
* improve logging
* simplify child policy state handling in picker
* use C++ style comments
* remove rls_config.proto, since it's not needed
* rewrote tests
* fix build
* fix copyright headers
* clang-format
* appease clang-tidy
* remove unnecessary dependendency on grpclb
* buildifier
* remove illegal term
* remove unneeded includes
* clang-format
* fix clang-tidy
* fix build
* move class declaration into .cc file
* clang-tidy again
* fix build
* reorganize code and misc cleanups
* clang-format
* fixed a bunch of asan bugs; still some left to debug
* fix asan problem
* make test work on IPv6-only machines
* move LRU list handling into Cache::Entry and add lock annotations
* use preincrement instead of postincrement
* fix deadlock
* add more FIXMEs
* clean up backoff timer logic
* fix build from merge
* clang-format
* fix include path to work on podspec builds
* clean up picker logic
* clang-format
* update rls.proto
* populate reason and stale_header_data in RLS request
* improve logging and fix some cache size type issues
* clang-format
* add some TODOs about a better way to expose channel creds to LB policies
* centralize handling of channel's default authority
* clang-format
* fix backup poller bug
* fix handling of call creds and authority on RLS channel
* remove unused params
* clang-format
* use two-phase update for child policies to avoid deadlocks
* fix portability issue
* minor cleanups
* update for change in grpc_error_get_status() API
* change test to store RLS requests and responses in proto form
* clang-format
* account for test slowdown factor in timeouts
* fix tsan failure and channelz linkage
* fix RLS authority death test
* fix test to not try to connect to target that should not work
* remove illegal term
* cacheSizeBytes is a required field
* add missing BUILD deps from merge
* add tests for connectivity state reporting and fix bugs found
* fix BUILD sanity
* buildifier
* fix BUILD package path
* fix bugs related to child policy lifetime and updates
* remove unnecessary srand() call from grpc_init()
* add test for two cache entries with the same target
* update rls.proto
* change RLS config parser test to support GRPC_ERROR_IS_ABSEIL_STATUS
* update upb codegen
* fix include path
* add env var guard
* avoid duplication of CountedService
* generate_projects
* add upper limit for cacheSizeBytes
* fix build
Co-authored-by: Muxi Yan <mxyan@google.com>
* make error independent
* Separate grpc_error into its own library
This is forward work to move Closure, ExecCtx into their own libraries
in order to make use of them in the activity code for resource quota
wakeups.
* Automated change: Fix sanity tests
* fixes
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* separate slice into internal and refcount, with refcount being the minimal base
* python3-ize
* Separate slices from mdelems for static data
Allows us to separate a small build target with just some slice basics
from the rest, which will help simplify the build system longer term.
Additionally exposed an opportunity to eliminate an init function.
Should also help in a month or so when it's time to jettison interned
metadata and slices entirely.
* fix compilation
* Automated change: Fix sanity tests
* fixes
* fixfixfix
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
* Add Python 3.10.0rc1 binary wheels
* Drop Python 3.5 artifacts
* Document the drop of 3.5
* Fix the wrong pip pointer
* Update manylinux2014 to a newer version, remove 3.5 distribtest
* Update manylinux aarch64 to see if the absl error go away
* Use the preferred alias
* Allow different wheel library to produce different tag order
* Remove unused shell var and log produced wheels
* Use copy instead of move
* Make bash happy about the wildcard
* Upgrade the debian image to use 3.5+ Python
* Polish the comments for the Dockerfiles
* Implement type safety for TLS
This is mostly free when compiler support is available, but requires
careful templating when implemented using pthread.
Significantly slimmed the tls.h interface; it now only defines the "TLS
keyword" for each supported compiler, delegating enforcement of correct
usage (i.e. must be static) to the compiler itself.
Implemented implicit conversion for the pthread wrapper so it can be
used (mostly) the same as native support. Notable exception to this is
that static_cast<void*> is needed when printing a pointer stored in TLS
as %p.
* Use GPR_THREAD_LOCAL macros consistently
* Bump Envoy and related submodules to latest
* Update googleapis
* Update scripts for newer submodules
* Update udpa
* Add opentelemetry
* Python changes for opentelemetry
* Update udpa in check_submodules
* Add opentelemetry to check submodules
* Regenerate upb files
* Add new proto dependency to upb-gen and Bazel
* Regenerate project
* Add isort_code.sh to sanity tests
* Run tools/distrib/isort_code.sh
* Fine tune the import order for relative imports
* Make pylint and project generation happy
* Fix a few corner cases
* Use --check instead of --diff
* The import order impacts test result somehow
* Make isort print diff and check output at the same time
* Let tools/run_tests/python_utils be firstparty library
* Run isort against latest HEAD
* fix: If client request iterators raise exceptions, _consume_request_iterator should not dump its stack to stderr
fixes: #25339
* fix: use format_exc
* fix: linter
* fix: lint round 2
This code adds an iomgr implementation that's backed by an EventEngine. This uses the EventEngine API alone, and separate work will introduce an EventEngine prototype to plug into it.
See also drfloob#1: @nicolasnoble has a pull request against this branch, implementing the libuv-based EventEngine. One goal here is to implement the iomgr code such that it can be merged independently without affecting normal builds.
This implementation can be built using bazel build --cxxopt='-DGRPC_USE_EVENT_ENGINE' :all
Some shortcuts are being taken to get a working, testable version of the engine. EventEngines are not pluggable, for example.
* support user provided audience in gdc and jwt
* fix 1st round of comments
* fix ruby and php to use new GDC API
* fix python clang issue
* address 2nd round of comments
* fix string_view issue
* remove length param in string_view constructor
* Handle gevent exception in gevent poller
Currently the gevent poller ignores exceptions raised by
`gevent.wait()`, which causes greenlets to be unkilable while waiting.
This change handles exceptions raised while waiting in the gevent
poller, cancels the gRPC call and propagates the error back to the
application.
Co-authored-by: Kostis Lolos <klolos@arrikto.com>
* Fix imports in header files
* Lint gevent tests
* Set grpc event type to GRPC_QUEUE_SHUTDOWN upon cancel error
To prevent `grpc_completion_queue_next()` to be called indefinitely when
the queue is shut down.
* Remove unnecessary `except *`
* Improve gevent tests
* Format code
* Remove unnecessary import
Co-authored-by: Kostis Lolos <klolos@arrikto.com>
* Add grpcio-csds pacakge
* Remove unused file
* Fix the proto import path issue
* Update the CSDS package and xds-protos for PY2
* Make tests happy
* Fix Bazel proto dependency
* Add Python2 tests for CSDS
* Expose code and details from context on the server side
This makes them accessible to server-side interceptors.
* Complete synchronous implementation
* Add experimental API notes
* Complete async implementation
* Revert bogus copyright change
* Fix metadata tests for py2.7
* Add new test to tests.json
* Check error codes in tests
* Bump autogenerated Python version
* pass XdsClient to LB policies via channel args
* add channel arg for overriding bootstrap config on a per-channel basis
* change tests to use new channel arg approach -- currently failing for server-side tests
* use the same channel args approach on the server side
* clang-format
* fix CircuitBreakingMultipleChannelsShareCallCounter test
* fix XdsEnabledServerTest test
* add TODO
* clang-format
* generate_projects
* fix clang-tidy
* fix build
* attempt to fix python
* rename ChannelData to ClientChannel
* make ClientChannel class definition public
* move retry code to its own filter
* move LB call factory method to ClientChannel class
* move dynamic termination filter out of ClientChannel class
* update comments
* remove retry parsing from client channel service config parser
* fix clang-tidy
* fix service_config_test
* clang-format
This caused problems internally. The interface deletion needs a cherrypick, but I will do an overnight test first, before I reintroduce. CC @nicolasnoble
* Implement the xDS Config Dump as CSDS in Core
* Revemp the logic of caching and constructing the CSDS response
* Unref created error
* Fix the clang tidy complains
* Resolve comments about symbol preload and Json ctor
* Improve readability
* Merge with new ADS logic && support latest CSDS
* Refactor the version/client_status logic
* Add support for error_state
* Add support for does_not_exist
* Add node information
* Fix the ProtoBuf segfault
* More test cases
* Refactor to use bytes as cache instead of JSON
* Apply clang_tidy's suggestion
* Resolve reviewer's requests
* Tiny stylish fixes && make sanity test happy
* WIP: still working on the matchers
* Update a bunch of matchers
* Improve readability a bit
* Resolve reviewer's suggestions
* Resolve reviewer's comments
* Make Unpack fail fast
* Revert "Revert "xDS status notifier (#25321)" (#25702)"
This reverts commit 3c9f3972e3.
* Remove connection from map when OnClose is not registered
* Reviewer comments
* Refactored with dynamic filters
* Error-tolerance tuned
* Fix leak of grpc_error and data race of canceller
* Adopt the latest xDS HTTP filter framework
* Fix fault injection tests' conflict with router filter
* Test alternative setup (override, no-override) without copy
* Refactor file strcutures of fault injection filter
* Rewrite the Json parsing/assembling logic again
* Added logic for aborting streaming RPC && resolve comments
* Revert "Revert "xDS HTTP filter support for gRPC client" (#25555)"
This reverts commit 785a128a83.
* xds_end2end_test: avoid flakes from lingering shutdown from previous test
* Download the working msys64 and use it to compile 27
* Remove the cygwin detection override logic
* Only install Python interpreters in jobs running on winserver2016
* Fix the batch script substring check
* Increase the timeout for related jobs
* TIL that there is an allow list for Kokoro env vars
Added logging for unsupported attributes
fixed make issues by moving grpc_authorization_engine under grpc
changed inet_aton to inet_pton
fixed issue leftover from initial implementation of evalargs
Merge remote-tracking branch 'upstream/master' into celeval_constructor
Iterate through policies and store CEL condition
git push origin ce Merge remote-tracking branch 'upstream/master' into celeval_constructor
added class and constructor comments for CelEvaluationEngine
CelEvaluationEngine class and constructor
Added cel_evaluation_engine to build
ran clang_tidy_code.sh
updated BUILD with cel_evaluation_engine
added rbac.upb.h/cc to BUILD
Rewrote cel_evaluation_engine code with upb conventions
removed unnecessary deps for cel_evaluation_engine in BUILD
Cel_evaluation_engine uses arenas to manage condition pointers
Used upb::Arena instead of upb_arena*, added temp_arena to save memory
Added action_allow_ member variable
Added fullstops to comments
initialize action_allow_
changed variable names to be more clear
removed unnecessary headers
correctly initialize the action_allow_ variable
changed constructor to use an initializer list
ran clang_tidy and clang_format scripts
added cel_engine_test
resolved merge conflicts and rewrote cel engine constructor to use new map api
changed variables to be more clear and made them const
changed syntax.upb.h so the build succeeds
changed syntax.upb.h so the build succeeds
changed version of upb in bazel_dpes
regenerated upb files with new upb version
added TODO regarding two-policy design
modified CelEvaluationEngine and tests to use two rbac policies
Made test messages consistent and added a case with too many policies
changed name from cel_engine to authorization_engine
fixed merge issues and updated authorization engine to v3 rbac
made constructor public and added namespace grpc_core
- Undo a bunch of hacks in src/boringssl/gen_build_yaml.py
- Store the structued data in YAML/dependencies.py so
we don't need to recreate the filters.
- Update setup.py accordingly
- Adds a new environment variable for turning on the build of ASM
for boring SSL.
- Only enables for x86_64 for now. I think this is likely the most
common target and the only machine I have readily accessible.
donnadionne