89519343a7
adding new bg read metrics to client and refactoring them into a new trace event ( #8493 )
2022-10-18 14:13:35 -05:00
1b2fcdd4f6
Merge pull request #8053 from sfc-gh-akejriwal/getsizetenant
...
Make the storage metrics functions tenant aware
2022-10-17 20:02:52 -07:00
6251e8be32
Track transaction cost for range reads
2022-10-17 10:08:49 -07:00
1c90fbbc61
blob: tenant verifyBlobRange() native api
2022-10-17 11:42:08 -05:00
4a33de31e9
blob: tenant listBlobbifiedRanges() native api
2022-10-17 11:41:45 -05:00
120b38a926
blob: tenant (un)blobbify() native api
2022-10-17 11:41:12 -05:00
0ee0ce62f0
blob: blobGranuleGetTenantEntry() for tenant and database calls
...
Currently, blobGranuleGetTenantEntry() expects to get the TenantName
through the transaction. Database calls cannot abide by this contract.
So enable the call to take in a TenantName and expect at least 1 to be
set.
2022-10-17 11:41:12 -05:00
84aa815026
Implemented first version of Transaction::getTotalCost, along with workload
2022-10-16 21:58:08 -07:00
0f9da9d1ad
Merge branch 'main' of github.com:apple/foundationdb into getsizetenant
2022-10-14 16:50:51 -07:00
854212fe94
Incorportate code review suggestions
2022-10-13 17:41:31 -07:00
be64b12ecf
Clean up TraceEvents, TODOs, and simplify workload.
2022-10-13 14:31:42 -07:00
df5825ff65
Merge pull request #8398 from sfc-gh-anoyes/anoyes/idempotency-id2
...
Initial work for automatic idempotency
2022-10-13 13:07:14 -07:00
293cc2a8fb
Each getRange request in a txn has a distinct trace id
...
Previously when multiple getRange requests are from the same parent
RL request, they share the same trace id from TransactionState, with
this change it has better granularity and makes it easier to compare
latency between getRange requests.
2022-10-12 12:12:58 -07:00
96574bacaa
Change Feed TSS Support ( #8384 )
...
* Change Feed TSS Support
* bug fixing for rare mismatch cases
* Adding rollback handling to tss change feed comparison
2022-10-12 10:23:51 -05:00
3223208f0c
Explicitly pass `arena` to the `withPrefix` function.
...
Also update some TraceEvents to be more succinct.
2022-10-11 18:35:01 -07:00
aad3899cb7
Enforce the lifetime of a write tx is actually <= MAX_WRITE_TRANSACTION_LIFE_VERSIONS
2022-10-11 13:46:39 -07:00
045452aec8
Add more detail to TraceEvents
2022-10-11 13:46:39 -07:00
6ef8d7ec95
Add AutomaticIdempotencyWorkload
2022-10-11 13:46:39 -07:00
f48d0d929b
Assign to idempotency id through transaction options
2022-10-11 13:46:39 -07:00
11658c7135
Merge branch 'main' of github.com:apple/foundationdb into getsizetenant
2022-10-11 13:09:47 -07:00
12aaabb283
Fix bugs in finding correct shards for tenant aware storage metrics.
...
* Account for tenant in during cache invalidation in NativeAPI actor.
* Account for tenant prefix while checking whether a shard is readable
on storage server.
This commit also adds many debug TraceEvents that will later be
removed.
2022-10-10 18:52:03 -07:00
b7ce834d28
Merge commit '7c89cd705faee52d5d78e6c77665cb7cc4502f58' into storageserver-pml
2022-10-07 11:39:42 -07:00
ea1325a552
Merge pull request #8319 from sfc-gh-tclinkenbeard/add-rare-code-probe-annotation
...
Add `rare` code probe decoration
2022-10-07 09:39:00 -06:00
3228afefd3
Unrevert #7578 - storage server PriorityMultiLock and PML rewrite.
2022-10-06 23:41:28 -07:00
f8186988bc
Merge pull request #8378 from sfc-gh-ajbeamon/tenant-support-for-database-switching
...
Add tenant support for database switching
2022-10-05 14:57:33 -07:00
e83229ed8b
- Version vector specific: Propagate the latest commit version, as ( #8393 )
...
part of the read request, when reading from a storage server.
2022-10-05 16:23:56 -04:00
dc917453c1
Targeted blob granules fault injection ( #8231 )
2022-10-05 13:44:38 -05:00
53593c77ac
When updating watches after a database switch, lookup the tenant again to get the ID on the new cluster. Update the switching test to set up tenants properly.
2022-10-03 09:24:13 -07:00
de90e1aab1
Improve the code in the actors and server, simplify the workload
2022-09-29 18:34:35 -07:00
f63934117d
Make the storage metrics function tenant aware [WIP]
...
This change makes the getEstimatedRangeSizeBytes function tenant aware.
Previously, this function would return the size of the requested
keyspace even if the tenant in the Transaction or DatabaseContext did
not match the tenant corresponding to the keyspace.
Also make some improvements to the new workload.
2022-09-29 18:32:04 -07:00
c8bb15e8ee
Merge pull request #8216 from sfc-gh-ajbeamon/backup-support-for-tenants
...
Include tenant and metacluster metadata in backups
2022-09-28 12:00:24 -07:00
985958c260
Add rare code probe decoration
2022-09-25 15:28:32 -07:00
5774249e5b
Revert "[DRAFT] Redwood PriorityMultiLock enable different launch limits to be specified based on different priority level."
2022-09-23 12:22:47 -06:00
f78eb8c778
Adding bg read amp metrics ( #8275 )
2022-09-22 16:31:27 -07:00
fda0d7223d
Update backup to include system key ranges needed for tenants. Run simulated backup tests with tenants.
2022-09-22 10:00:13 -07:00
04b4960786
Merge branch 'main' into fzhao/RedwoodIOLaunchLimit
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/ReadWrite.actor.cpp
2022-09-22 00:39:51 -07:00
7a09b701cc
Merge pull request #8141 from sfc-gh-jfu/network-disable-bypass
...
Introduce network option for disabling mvc bypass
2022-09-21 17:33:48 -07:00
4ea4546cb6
blob/java: verifyBlobRange() with latestVersion
2022-09-21 14:07:16 -07:00
e353169a50
blob: teach purge about latestVersion
...
This teaches purgeBlobGranules about latestVersion and rejects
versions <= 0.
2022-09-21 14:04:58 -07:00
4e8ac6c811
remove network option on native side since option is no longer being forwarded
2022-09-20 11:10:12 -07:00
1f778f9d76
Merge branch 'main' of github.com:apple/foundationdb into network-disable-bypass
2022-09-20 09:30:53 -07:00
0f3f493c28
Merge pull request #8218 from sfc-gh-jslocum/aligned_purge
...
fixes for non-aligned blob range calls
2022-09-19 15:00:02 -05:00
4fd64630e8
Convert literal string ref instances to use _sr suffix
2022-09-19 11:35:58 -07:00
88f88707f5
fixes for non-aligned blob range calls
2022-09-16 19:06:15 -05:00
1abac8ea9f
check on shared state ptr in native api and add to test spec in api tester
2022-09-16 15:11:33 -07:00
7b4598a53d
Merge pull request #8197 from sfc-gh-jslocum/bg_code_coverage_cleanup
...
cleaning tss and blob granule file code probes
2022-09-16 09:32:20 -07:00
4ead9a697f
cleaning tss and blob granule file code probes
2022-09-16 09:51:33 -05:00
977c03ff78
not counting end of stream as an error for change feeds
2022-09-15 17:37:52 -05:00
ac65c3f569
merge upstream main
2022-09-15 14:19:19 -07:00
7b0285a205
attempt to pass sim tests and api tests
2022-09-15 11:02:28 -07:00
82adc1e856
Make g_simulator a pointer
2022-09-15 09:00:33 -07:00
a7956f42a4
add version guard and add network option to testConfig
2022-09-14 13:15:24 -07:00
3e5e49b635
Operational improvements to limit thundering herd effect of many change feed queries being retried simultaneously
2022-09-14 09:57:21 -05:00
e5eabbf3df
Additional observability for change feeds
2022-09-14 09:55:15 -05:00
ee99de7cf3
Merge branch 'apple:main' into RedwoodIOLaunchLimit
2022-09-12 10:58:12 -07:00
f6d6aa719f
initial commit to introduce network option for disabling mvc bypass
2022-09-09 14:29:38 -07:00
006fc9a0ef
code formatting
2022-09-09 11:58:36 -07:00
75a096a5e5
Merge branch 'main' of github.com:apple/foundationdb into jfu-tenant-special-key-space
2022-09-09 10:12:19 -07:00
726d5215a0
Remove API 720 guards for tenants (experimental feature) and the cluster ID special keys ( #8108 )
...
* Remove API 720 guards for tenants (experimental feature) and the cluster ID special keys (no need to guard)
* Enable the relaxed special key access in transactions that need to use special key-space APIs introduced in 7.2
2022-09-08 17:22:36 +02:00
b6fb1034a9
Fix crashes on database create errors ( #8115 )
...
* ApiTester: enable access to database instances in workloads
* ApiTester: Inject database create errors
* Fix getClusterProtocol and database shared object initialization in case of database create errors
* Check deferred error in all Thread-Safe API calls returning a future
* ApiTester: disable injection of database create errors for tests with old versions
2022-09-08 09:21:51 +02:00
e836b743c0
Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/grvError
2022-09-07 20:50:03 -07:00
7fc4a92d09
Merge pull request #8089 from sfc-gh-jslocum/unblobbify_idempotent
...
Blob range idempotency test and fixes
2022-09-07 17:31:05 -07:00
da7ce5231c
Merge branch 'main' of github.com:apple/foundationdb into jfu-tenant-special-key-space
2022-09-07 13:30:08 -07:00
255795eac0
update test workload and reserve space for vector copies
2022-09-07 13:25:38 -07:00
9fe12a256b
catch grv_proxy_memory_limit_exceeded in getConsistentReadVersion
2022-09-06 22:18:14 -07:00
dbb6357371
add conflict range tests and change tenant prefix code to work with RYW
2022-09-06 16:55:57 -07:00
b2ae592b51
Merge pull request #8082 from sfc-gh-vgasiunas/vgasiunas-api-version
...
ApiVersion: Small cleanups
2022-09-06 10:03:45 -07:00
105c4e5236
Merge branch 'apple:main' into RedwoodIOLaunchLimit
2022-09-06 09:38:06 -07:00
91219a4a28
sync
2022-09-06 09:37:00 -07:00
8ca8691837
ApiVersion: Minor clean ups
2022-09-05 10:42:42 +02:00
3ce737b2fa
Merge pull request #8086 from sfc-gh-tclinkenbeard/mark-promisestream-send-nonconst
...
Mark `PromiseStream::send` non-const
2022-09-04 14:55:47 -07:00
94e0c83214
Blob range idempotency test and fixes
2022-09-02 15:53:06 -05:00
3dec9749d3
Mark PromiseStream::send non-const
2022-09-02 10:41:56 -07:00
2251ae0fc2
Added summarize blob granules c api and tests ( #8076 )
...
* Added summarize blob granules c api and tests
* addressing review comments
* format
2022-09-02 17:08:51 +02:00
80a0816157
flow: switch from hard coded to ApiVersion like ProtocolVersion ( #8071 )
...
* flow: add ApiVersion to replace hard coding api version
Instead of hard coding api value, let's rely on feature versions akin to
ProtocolVersion.
* ApiVersion: remove use of -1 for latest and use LATEST_VERSION
2022-09-02 09:28:13 +02:00
c7f7544231
Merge remote-tracking branch 'upstream/main' into RedwoodIOLaunchLimit
2022-09-01 14:53:11 -07:00
61233123b2
format
2022-09-01 11:26:32 -07:00
663b1b9711
Handling request_maybe_delivered in reply promise stream and injecting faults to make sure the code path is exercised
2022-09-01 12:57:09 -05:00
0aa096dc17
sync with upstream main
2022-08-31 15:46:39 -07:00
046141b5be
blob: only allow unblobbification on aligned to ranges
...
If unblobbify request comes in, reject it if the start and end do not
align to a blob granule boundary.
2022-08-31 09:23:33 -07:00
dc27c9bbd7
blob: add basic pagination to listBlobbifiedRanges()
2022-08-31 09:23:33 -07:00
557efa1b53
Merge pull request #7406 from sfc-gh-fzhao/RedwoodDataMovementNonCacheRead
...
Redwood add non-cache reads options
2022-08-30 17:25:05 -07:00
269d54b7ad
One additional case for new error, and dropping retry limit for upgrade test back down ( #8041 )
2022-08-30 11:01:18 -05:00
d90cdcc498
Adding new retryable error type for blob granule requests ( #8023 )
2022-08-30 15:07:10 +02:00
5f9d8e94af
Adding future_version reply to change feed operations for behind ss ( #8000 )
2022-08-29 17:20:23 -05:00
f257b6c2d0
- Consider logging a trace event in "DatabaseContext::getLatestCommitVersions" ( #8008 )
...
only if the version vector doesn't have an entry for a tag.
- Address a bug in "run_custom_cluster.sh" (Note: code change provided
by Dan).
2022-08-29 09:22:34 -07:00
8314e80371
Fixed a few bugs which caused ratekeeper to unnecessarily throttle a cluster ( #8006 )
...
* do not count recently created change feeds for throttling
* fix: blocked assignments were not decremented when force purging
* fix: created needs to be updated when the changefeed is reset
* added asserts to detect if ratekeeper is throttled on blob workers
2022-08-26 15:38:31 -07:00
46b02cab49
Blob granule summary implementation (in native client) ( #7981 )
...
* implemented blob granule summary call in native client
* clean up prints
2022-08-26 14:04:59 -05:00
337a2f8130
try to account for errors in fuzzapicorrectness and add test case in special key test
2022-08-25 15:28:06 -07:00
ee707b8f87
initial commit for explicit tenant support in special key space
2022-08-25 12:44:10 -07:00
e3c46ffeef
change FDB_DEFINE_BOOLEAN_PARAM
2022-08-24 15:53:05 -07:00
7c6dbaf3cf
update trState ReadOption to optional
2022-08-24 15:04:17 -07:00
e04e3885b9
Adding BlobRange test ( #7868 )
...
* Adding BlobRange test
* refactored blobrange test to use new db functions, and fixed several bugs
* bug fixes for blob manager and verifyBlobRange
* More range unaligned fixes
* cleaning up test and disabling tests that don't work yet for now
* removing overzealous assert in blob manager
* more fixes for overzealous assert
* cleanup and renaming test
* adding chaos to blob ranges test
2022-08-24 11:30:37 -05:00
4e748d6bed
use optional ReadOptions and RangeReadOptions in requests
2022-08-23 17:16:47 -07:00
493771b6a8
Throttle the cluster if the blob manager cannot assign ranges ( #7900 )
...
* Throttle the cluster if the blob manager cannot assign ranges
* fixed a number of different bugs which caused ratekeeper to throttle to zero because of blob worker lag
* fix: do not mark an assignment as block if it is cancelled
* remove asserts to merge bug fixes
* fix formatting
* restored old control flow to storage updater
* storage updater did not throw errors
* disable buggify to see if it fixes CI
2022-08-23 13:33:46 -05:00
98a7ec1797
Blob Granules Cleanup ( #7941 )
...
* Cleaned up BlobGranule TODO + FIXMEs and addressed some
* popping feed at correct version
* blob worker taking over a granule will pop from where previous worker left off
* addressed fixme of blob worker not re-snapshotting from old change feed
* formatting
* more change feed popped fixes after pop updates
* Getting rid of change feed parallelism lock since it can cause deadlocks in fetching, and relying on full fetch lock
* New blob worker metric and fixing old one
* server-side popped checking still doesn't work because of pops at non-mutation versions
* format
2022-08-19 17:25:31 -07:00
e69e93496b
inherit readOptions passed from FetchKeys
2022-08-17 14:05:32 -07:00
3d400cff64
blob: verifyBlobRange() c api
2022-08-16 13:29:23 -07:00
1c2109dcbd
blob: add rangeLimit to getBlobGranuleRanges()
2022-08-16 13:29:23 -07:00
96f3dd67b0
blob: add listBlobbifiedRanges() api
2022-08-16 13:29:20 -07:00
07018eefe5
update parameter order
2022-08-16 13:16:38 -07:00
5085c21d0d
blob: check if blob range exists and fail accordingly
2022-08-15 16:25:36 -07:00
736ef4f2c9
blob: move blobrange command implementation to native api
2022-08-15 16:25:36 -07:00
eb16947860
Merge pull request #7844 from jzhou77/grv-error
...
Buggify GRV proxy to return errors and fix bugs found
2022-08-15 13:59:21 -06:00
3fed0456ca
Add an verify option for \xff\xff/worker_interfaces special keys ( #7873 )
...
* Add the verify option for \xff\xff/worker_interfaces
* Remove unused code
* update documentations
* update documentations
* solve comments from review
* update some of the comments to be more clear
2022-08-15 14:05:07 -04:00
a9d3c9f9b3
Added throttling when a blob worker falls behind ( #7751 )
...
* throttle the cluster when blob workers fall behind
* do not throttle on blob workers if they are not enabled
* remove an unnecessary actor
* fixed a compile error
* fetch blob worker metrics at the same interval as the rate is updated, avoid fetching the complete blob worker list too frequently
* fixed another compilation bug
* added a 5 second delay before bw throttling to prevent false positives caused by the 100e6 version jump during recovery. Lower the throttling thresholds to react much quicker to bw lag.
* fixed a number of problems
* changed the minBlobVersionRequest to look at storage server versions since this will be a lot more efficient
* fix: do not let desired go backwards
* fix: track the version of notAtLatest changefeeds for throttling
* ratekeeper now throttled blob workers by estimating the transaction per second throughput of the blob workers
* added metrics for blob worker change feeds
* added a knob to disable bw throttling
* fixed the transaction options in blob manager
2022-08-12 13:15:56 -07:00
8fb6d59e94
Use error_code_grv_proxy_memory_limit_exceeded
...
instead of error_code_proxy_memory_limit_exceeded
2022-08-12 11:36:17 -07:00
ed1b7ef173
Add a client knob GRV_ERROR_RETRY_DELAY
2022-08-12 11:13:32 -07:00
317866fbbf
Add retry in waitForCommittedVersion for GRV error response
2022-08-12 11:13:32 -07:00
970105f318
Client needs to handle proxy_memory_limit_exceeded error
2022-08-12 11:13:32 -07:00
0326d53965
Split proxy_memory_limit_exceeded to commit/grv specific exceptions
...
Currently GRV is reporting proxy_memory_limit_exceeded error which has
error message claiming Commit proxy failing. This split should remove
such confusion.
2022-08-12 00:45:57 -07:00
7c155f4521
Granule force purging ( #7846 )
...
* Granule purge cannot delete history entry for fully deleting granule until all children are completely done splitting
* Several purging fixes related to granule history
* Fixed typo in refactor
* fixing memory model for purgeRange
* formatting
* weakening granule purge test for now
* cleanup
* First version of force purging granules
* fixing issue in BW range assignment reporting
* Fixing incorrect assert with force purging
* Error handling when checking force purged state
* fixed force purging and recover/reassign range races and check
* Handling force purge + boundary change race
* more places to check for force purged status
* fixed manager restart in the middle of force purge bug
* fixing same-BM purge and assignment races in all cases
* weakening orphaned granule history check a bit because of difficult to solve races
* fixing txn options on retry
* loading force purged ranges at start to avoid resuming a merge that is being force purged
* cleanup
* Enabling purging in granule tests, and adding check for leaked change feeds in force purge
* formatting
* missed parameter in merge conflicts
* Fixing leaked change feed race with merge and force purge
* adding change feed cleanup when new blob manager recovers in-progress merge that raced with force purge
* added forcepurge fdbcli command
2022-08-11 15:22:32 -07:00
28469196b5
add comments
2022-08-10 15:45:15 -07:00
d24a52952c
update readOption struct
2022-08-10 10:16:26 -07:00
735b78cd0c
Several Assorted Blob Granule/Change Feed bug fixes ( #7842 )
...
* fixing another actor cancelled callback issue in change feed stream
* remove wait for on move, to ensure stream can't read from a shard the SS no longer owns
* fixing key alignment after downsampling split
* Fixing out-of-date granule reporting itself as a merge candidate
* fix formatting
2022-08-10 08:28:09 -07:00
1a5b8fbd30
resolve conflicts
2022-08-09 09:57:11 -07:00
e4fb565057
use readOptions to pass type and cacheResult
2022-08-08 15:30:56 -07:00
02b69e3c66
Merge pull request #7806 from sfc-gh-ajbeamon/fix-unknown-tenant-handling
...
When a transaction operation gets an unknown tenant error, it needs to reset the tenantID
2022-08-08 10:33:10 -06:00
79571dd2b4
Testing upgrades to a future version of FDB ( #7780 )
...
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli
* Auto format python files used in upgrade tests
* Add a test for upgrading to a future FDB version
* Emphasize that the options for using future protocol version are intended for test purposes only
* Make the global variable for current protocol version visible only locally
* Refactirng to avoid using currentProtocolVersion() in static intialization
* Update go bindings
2022-08-08 17:29:49 +02:00
17858cf8a0
When a transaction operation gets an unknown tenant error, it needs to reset the tenant ID so it can be updated in the next tenant lookup request.
2022-08-06 13:13:46 -07:00
ddbf32d35e
adding knob and fixing blobrange cli check bug with too many granules ( #7793 )
2022-08-05 11:19:17 -05:00
d5c3679046
merge upstream main and resolve conflicts
2022-08-04 12:15:00 -07:00
dccf0c5fd7
add readType to other requests
2022-08-02 15:47:15 -07:00
1b693a588a
Merge branch 'main' into feature-metacluster
2022-08-01 14:43:14 -07:00
ea76eb2beb
Transactions could change tenant ID part way through. The dummy transaction could set a new tenant ID but use the old tenant prefix.
2022-07-30 16:16:18 -07:00
e8e4f3ad3a
Merge branch 'main' into feature-metacluster
...
# Conflicts:
# fdbclient/include/fdbclient/Tenant.h
2022-07-28 16:53:29 -07:00
83d23512d2
Track when version not found in version vector. ( #7653 )
...
* Track when version not found in version vector.
* Respond to comments
* include vv in event
* Respond to review comments.
* Fix bug where we logged too many lines
2022-07-28 18:26:15 -04:00
c6342a6e5b
Merge branch 'main' into features/authz
2022-07-27 20:51:32 +02:00
27e2bf60c6
Handle case where exception is thrown before tenant prefix is prepended
2022-07-27 20:13:25 +02:00
7c6b3fb0b8
Merge branch 'main' into feature-metacluster
2022-07-27 08:55:10 -07:00
77956dc7ae
Merge pull request #7639 from sfc-gh-jslocum/cf_metadata_rewrite
...
Change Feed Metadata Rewrite and adding targeted fault injection
2022-07-26 18:10:37 -05:00
5169616b16
Fix unresolved merge conflicts
2022-07-27 00:38:16 +02:00
e2a3fedfc7
Merge branch 'main' into features/authz
2022-07-27 00:08:57 +02:00
8b5b7a9b92
Merge pull request #7689 from sfc-gh-jslocum/purge_fixes
...
Granule Purge Fixes
2022-07-26 17:02:24 -05:00
ba31229797
addressing review comments
2022-07-26 16:37:06 -05:00
6f9c87f2a5
Move dummy transaction flag to TransactionState
...
Also add more description of what the flag does and when it's set
2022-07-26 23:27:46 +02:00
4c4eb3fc48
Merge pull request #7704 from sfc-gh-ajbeamon/add-comments-to-tenant-function
...
Add comments explaining the use of the TransactionState tenant() and hasTenant() functions
2022-07-26 14:01:50 -07:00
adc601e8a3
Simplify tenant-aware dummy transaction
2022-07-26 21:30:44 +02:00
d31324102d
Add comments explaining the use of the TransactionState tenant() and hasTenant() functions
2022-07-26 10:17:57 -07:00
32ab7421e1
add readtype to requests
2022-07-26 10:12:54 -07:00
7c2ed53aef
Fix: during recovery, it was possible for tenant operations to briefly fail because the tenant mode is not known
2022-07-25 13:03:15 -07:00
553e6aba75
Make dummy transaction commits tenant-aware
...
Dummy transaction commits are made when earlier transaction returned
commit_unknown_result or request_maybe_delivered, to affirm the status
of earlier transaction. Without tenant-awareness, these transactions
will fail due to lack of authorization in case of untrusted clients.
2022-07-25 18:20:37 +02:00
ee1b0cdd43
making purge failures fail test
2022-07-23 13:10:03 -05:00
41c1bef3bf
Merge branch 'feature-tenant-groups' into feature-metacluster
2022-07-22 14:26:02 -07:00
25d9fa856c
fixup! Update code per comments
2022-07-21 23:14:31 -07:00
32b4968835
Add TraceEvent to the construction/destruction of DatabaseContext
...
This is to figure out why huge databaseLogger related TraceEvents
generated after 7.1 released.
2022-07-21 19:38:59 -07:00
3e1763e7ac
Merge branch 'main' into feature-metacluster
2022-07-21 14:48:53 -07:00
410f27412b
Merge pull request #7620 from sfc-gh-ajbeamon/make-tuple
...
Add a Tuple::makeTuple function to easily construct a tuple
2022-07-20 17:09:10 -07:00
fee131f24d
Merge pull request #7591 from sfc-gh-ajbeamon/key-backed-types-range-read-more
...
Add support for returning a more flag from key backed types' range reads
2022-07-20 15:57:50 -07:00
279296c29f
Merge branch 'tenant-metadata-change' into feature-metacluster
...
# Conflicts:
# fdbclient/SystemData.cpp
# fdbclient/Tenant.cpp
# fdbclient/include/fdbclient/SystemData.h
# fdbclient/include/fdbclient/Tenant.h
# fdbclient/include/fdbclient/TenantManagement.actor.h
# fdbserver/TenantCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-20 09:18:27 -07:00
fd9201f60b
Merge branch 'main' into cf_metadata_rewrite
2022-07-20 07:55:00 -05:00
537ceff8ac
Remove the ability to configure a tenant subspace. Rename the prefixes used for tenant metadata.
2022-07-19 14:32:05 -07:00
190ad8c7e9
Convert existing tuple usages to use Tuple::makeTuple()
2022-07-19 13:45:59 -07:00
1de37afd52
Make TEST macros C++ only ( #7558 )
...
* proof of concept
* use code-probe instead of test
* code probe working on gcc
* code probe implemented
* renamed TestProbe to CodeProbe
* fixed refactoring typo
* support filtered output
* print probes at end of simulation
* fix missed probes print
* fix deduplication
* Fix refactoring issues
* revert bad refactor
* make sure file paths are relative
* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
3ceb2a0639
Merge branch 'main' into feature-metacluster
2022-07-19 13:04:54 -07:00
2322730345
fix issues related to terminate storage server
2022-07-19 11:19:30 -07:00
775ab3323e
update tag client cache on getKeyLocation ( #7613 )
2022-07-19 11:24:17 -04:00
2573c0018c
cleanup after passing tests
2022-07-19 08:00:25 -05:00
4540e2879c
Change Feed Metadata refactor to fix races, simplify model, and clean up code
2022-07-18 12:46:11 -05:00
ca29cd9f41
Make RequestStreams accept empty TenantInfo
...
+ Make Authz tokens survive Transaction resets
+ Make tokens explicitly resettable by assigning empty value to authz token TR option
+ Fill out missing trace for some failed token verifications
+ Trace non-sensitive token parts upon failed verification
2022-07-18 13:30:55 +02:00
312e160a12
use PriorityMultiLock in storage server
2022-07-14 15:29:54 -07:00
30afc35439
Merge branch 'key-backed-types-range-read-more' into feature-metacluster
2022-07-13 16:00:01 -07:00
91949439d2
Add support for returning whether a key-backed type range read has more results or not.
2022-07-13 15:46:10 -07:00
cb499fa5db
Merge branch 'main' into feature-metacluster
2022-07-13 15:28:34 -07:00
03a496c0e0
Make auth token survive transaction resets
2022-07-13 20:34:01 +02:00
0b0ac16a4c
Merge branch 'main' into granule_merging
2022-07-12 09:09:30 -05:00
2edbcf2c65
Merge pull request #44 from apple/main
...
Merge main
2022-07-12 07:51:22 -06:00
58aa5a9e1d
Merge pull request #7539 from sfc-gh-fzhao/SetFetchKey
...
set isFetchKeys for getRangeStreamFragment
2022-07-11 13:25:24 -07:00
dbf8f98481
Apply Clang format
2022-07-11 17:10:45 +02:00
61f3d14ec6
Add TokenCache test
...
Add function authz::jwt::TokenRef::toStringRef() for token tracing
Add StringRef::toStringView()
2022-07-11 13:42:41 +02:00
2ae17233d8
addressed review comments
2022-07-08 11:19:14 -06:00
f268265d96
Apply suggestions from code review
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
Co-authored-by: Junhyun Shim <junhyun.shim@snowflake.com>
2022-07-08 10:42:44 -06:00
bc5bfaffda
Shard based move ( #6981 )
...
* Shard based move.
* Clean up.
* Clear results on retry in getInitialDataDistribution.
* Remove assertion on SHARD_ENCODE_LOCATION_METADATA for compatibility.
* Resolved comments.
Co-authored-by: He Liu <heliu@apple.com>
2022-07-07 20:49:16 -07:00
207082f03e
comment out setting isFetchKeys
2022-07-07 12:55:25 -07:00
6e7603aa35
set isFetchKeys for getRangeStreamFragment
2022-07-07 11:14:11 -07:00
1519f24f77
Merge branch 'main' into feature-metacluster
2022-07-07 09:35:40 -07:00
c4b0f6eaae
Add an internal C API to support connection to a cluster using a connection string ( #7438 )
...
* Add an internal C API to support memory connection records
* Track shared state in the client using a unique and immutable cluster ID from the cluster
* Add missing code to store the clusterId in the database state object
* Update some arguments to pass by const&
2022-07-07 10:12:49 +02:00
56541167c2
send token with tenant info object
2022-07-06 20:58:17 -06:00
c9b553663e
Merge branch 'main' into feature-metacluster
...
# Conflicts:
# bindings/java/src/main/com/apple/foundationdb/TenantManagement.java
# fdbcli/TenantCommands.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/TenantSpecialKeys.actor.cpp
# fdbclient/include/fdbclient/KeyBackedTypes.h
# fdbclient/include/fdbclient/RunTransaction.actor.h
# fdbclient/include/fdbclient/SpecialKeySpace.actor.h
# fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-06 15:44:21 -07:00
1e8225a19e
Merge pull request #7511 from sfc-gh-ajbeamon/add-subrange-support-to-tenant-special-keys
...
Update the tenant special keys submodule to support multiple sub-ranges
2022-07-06 10:33:14 -07:00
2eae95aabe
Merge branch 'main' into feature-metacluster
2022-07-01 14:07:34 -07:00
037c06a688
Merge remote-tracking branch 'apple-upstream/main'
2022-07-01 04:51:17 +00:00
aea4d802c6
Merge branch 'main' into feature-metacluster
2022-06-30 15:08:09 -07:00
2f67328a0c
Update the tenant special keys submodule to support multiple sub-ranges. This will enable future work that allows configuring tenants at the same time as creating them.
2022-06-30 15:03:37 -07:00
48bcf6eaba
parameterizing min split size instead of hardcoding it to min_shard_bytes ( #7404 )
2022-06-30 12:28:22 -07:00
c74710f6d5
- Address a review comment
...
- Optiize out the code in DatabaseContext::getLatestCommitVersions()
in the case where the feature is turned off.
2022-06-29 10:13:08 +00:00
e1a93988ef
Merge branch 'main' into feature-metacluster
2022-06-28 14:58:07 -07:00
9d16260822
Merge remote-tracking branch 'origin/main' into features/authz
2022-06-28 13:13:51 -06:00
086e4bff06
Merge remote-tracking branch 'origin/main' into global-tag-throttling3
2022-06-28 10:18:13 -07:00
0da13a7797
- Optimize out the version vector specific code (on the client)
...
when the version vector feature is disabled.
2022-06-28 09:17:53 +00:00
4452e53342
Fix a problem caused by missing state.
2022-06-27 23:52:54 -04:00
62d4ac550a
Merge branch 'main' of https://github.com/apple/foundationdb into feature/ddtxn
2022-06-27 11:35:22 -07:00
3aaae9c521
Merge remote-tracking branch 'origin/main' into features/authz
2022-06-27 11:07:14 -06:00
8af056e7b0
fdbclient now compiling
2022-06-23 18:05:36 -06:00
ffaf15c12a
moved wellknownendpoints and fixed some includes
2022-06-23 17:03:53 -06:00
a5054b2beb
move getServerListAndProcessClasses to NativeAPI
2022-06-23 15:28:45 -07:00
d35445a868
enforce include modularization in cmake
2022-06-23 14:37:35 -06:00
9f3819752f
Change the command to create a metacluster from using 'configure tenant_mode=management' to 'metacluster create <NAME>'. Distribute this name to all processes in a metacluster. Eliminate the tenant mode entirely from metacluster clusters, instead relying on a metacluster registration key.
2022-06-22 12:15:43 -07:00
2391e58fb2
Merge remote-tracking branch 'origin/main' into global-tag-throttling3
2022-06-21 10:09:15 -07:00
bcd8785767
Remove old debug tracing
2022-06-20 00:59:22 -07:00
34e6a8f942
Merge pull request #7399 from sfc-gh-jslocum/bg_tenant_improvements
...
Bg tenant improvements
2022-06-17 11:19:41 -05:00
1cc466e068
fixes and review comments
2022-06-17 08:17:44 -05:00
b3597ef3a8
Added plumbing for tenant-aware purge granules
2022-06-16 13:04:34 -05:00
13d8b13722
Migrated Authz code to use JWT
2022-06-13 18:20:27 -06:00
799fe32346
Merge remote-tracking branch 'origin/main' into features/authz
2022-06-13 18:02:11 -06:00
207e0bc105
Fix a few places we weren't doing exponential backoff ( #7349 )
...
* Fix a few places we weren't doing exponential backoff
We re-create the transaction every iteration of each of these retry
loops, so we need to manage exponential backoff here ourselves.
Closes #7301
* Remove former Backoff definition
2022-06-13 13:18:58 -07:00
df71a49bf6
Merge remote-tracking branch 'origin/main' into global-tag-throttling3
2022-06-13 10:03:10 -07:00
75c858eb2c
Differentiate between different quotas in GlobalTagThrottling workload
2022-06-13 10:01:55 -07:00
710612442c
small fixes
2022-06-09 17:21:21 -05:00
d6920cde28
Implemented blob granule merging
2022-06-09 10:50:53 -05:00
a3289f9cab
adding tenant prefix to bg ranges call
2022-06-06 14:09:10 -05:00
2907aafa60
Merge branch 'main' into feature-metacluster
2022-06-02 12:18:42 -07:00
8cc51064a5
Add NAPI: prefix to VerifyCausalReadRisky span. ( #7285 )
2022-06-01 11:47:28 -05:00
29ebcdbfa9
Merge remote-tracking branch 'origin/main' into global-tag-throttling3
2022-05-31 14:53:51 -07:00
4d6ec7663f
Close file at the end of runNetwork rather than in stopNetwork in order to avoid a race condition,
...
also in order to continue tracing the pending network thread activity.
Poll event throttler only in the network thread in order to avoid a race condition.
2022-05-31 13:59:09 +02:00
4c07064632
Fix reference cycle in GlobalConfig
2022-05-27 10:25:29 -07:00
cd2a575e02
Move the resolve of coordinator hostname from getCoordinatorProtocol() to getClusterProtocolImpl(). ( #7245 )
...
* Move the resolve of coordinator hostname from getCoordinatorProtocol() to getClusterProtocolImpl().
* Guard DNS Cache behind a knob.
2022-05-26 09:45:54 -07:00
458e8018e1
review comments
2022-05-25 17:16:56 -05:00
fd7b662767
cleaning up
2022-05-25 17:16:56 -05:00
dd637dd6a7
passes tenant simulation test
2022-05-25 17:16:56 -05:00
e3a570d50d
Blob worker tenant integration mostly works
2022-05-25 17:16:56 -05:00
4015b5b2a0
Several Blob Granule bug fixes found in nightlies ( #7170 )
...
* Fixing simulation validation assert that was tripping incorrectly
* Commenting out debugging prints
* Fixed multiple error propagation issues in blob worker
2022-05-23 10:07:16 -07:00
16cf087667
Allow fetching invalid tenant id for getKeyLocation
2022-05-23 10:00:27 -06:00
d784173f7f
Add first implementation of tenant creation and deletion in a metacluster
2022-05-20 15:21:21 -07:00
5e8c3130e6
Extend the reboot interface to accept comma-delimited string;
...
Only fetch requested worker interfaces in rebootWorkerActor
Add killall test to multiprocess_fdbcli_tests
Make the rebootWorker only sends requests if all the requested workers are valid
Add comments; Organize print texts; Fix the bug clear the map every time run kill/suspend/expensive_data_check
2022-05-20 12:06:33 -07:00
61490022b4
Enabled checkpoint, restore test for RocksDB format.
2022-05-19 13:27:21 -07:00
69261f9f10
Merge branch 'main' into feature-metacluster
...
# Conflicts:
# fdbclient/MultiVersionTransaction.actor.cpp
# fdbclient/MultiVersionTransaction.h
# fdbserver/tester.actor.cpp
2022-05-19 09:11:30 -07:00
993a79b42c
Restart protocol monitoring logic when we cycle the coordinator, even if its the same coordinator. Add some defensive code in case we fetch the protocol version for an absent peer. ( #6397 )
2022-05-19 10:00:23 +02:00
f00606fd71
Enable MATCHED and UNMATCHED mode for index prefetch ( #7162 )
...
* Enable MATCHED and UNMATCHED mode for index prefetch
MATCHED mode returns index entries whose secondary KVs are present,
UNMATCHED mode returns index entries whose secondary KVs are absent.
Note that the conflict read range of this txn is set in 2 steps:
* Set the conflict range for primary query according to request
* Set the conflict ranges for secondary queries according to responses.
As a result, conflicts of different match_index mode are taken care of.
* Fix c binding
2022-05-18 09:16:28 -07:00
e8e26c9f7c
Remove the internal special key-space mechanism (at least for now), instead relying on the local logic to implement the metacluster functions. Add ability to force remove a cluster. Improve idempotency of register operation.
2022-05-18 08:18:25 -07:00
853e6a346b
Optimization: support removing index conditionally ( #7116 )
2022-05-13 10:10:43 -07:00
58c818985d
Apply clang-format to several files
2022-05-10 11:21:45 -07:00
051bd3102d
Revert bad merge change
2022-05-09 14:54:51 -07:00
f88cbf9309
Address review comments
2022-05-09 14:54:51 -07:00
f14baf2af8
clang-format changes
2022-05-09 14:54:51 -07:00
9173e2e19b
Move GlobalConfig to DatabaseContext
2022-05-09 14:54:51 -07:00
7578d5ebc7
Create GlobalConfig object for each database instance
...
Currently, GlobalConfig is a singleton that means for each process there is only
one GlobalConfig object. This is bug from clients perspective as a client can
keep connections to several databases. This patch tracks GlobalConfig for each
database using an unordered_map in flowGlobals.
We discovered this bug while testing multi-version client, where the client got
stuck. This was lucky, as normally it'd just write down config to the wrong
database.
2022-05-09 14:54:51 -07:00
d6ac3e57a4
Add internal metacluster special key-space modules
2022-05-07 20:14:03 -07:00
c9a5b6d789
Avoid re-adding transaction tags in ReadWriteWorkload
2022-05-07 15:58:05 -07:00
154ae7559e
- If getRange() is called with "latestVersion" as the read version
...
then both GetKeyValuesFamilyRequest and getRangeFallback should do
reads at the same version.
2022-05-06 18:10:37 +00:00
7c3b99f5bd
Add support for connection to clusters via a connection string. Start work on multi-cluster operations with data cluster registration.
2022-05-06 09:53:53 -07:00
0c41225d76
Add support for setting and storing the tenant group for a tenant
2022-05-05 08:57:01 -07:00
2102ed1eaa
- Remove "stale_version_vector" error code.
2022-05-03 21:56:11 +00:00
7e860dc92c
Some initial metacluster implementation
2022-05-03 12:53:35 -07:00
484558250b
- Do not consult version vector if the client has obtained the read
...
version from its GRV cache.
2022-05-03 19:23:37 +00:00
dc9e782ccc
OpenTelemetry Tracing Perf Fixes ( #6990 )
2022-05-02 14:56:51 -05:00
c69a07a858
Check in the new Hostname logic. ( #6926 )
...
* Revert #6655 .
20220407-031010-renxuan-c101052c21da8346 compressed=True data_size=31004844 duration=4310801 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=1:04:15 sanity=False started=100047 stopped=20220407-041425 submitted=20220407-031010 timeout=5400 username=renxuan
* Revert #6271 .
20220407-051532-renxuan-470f0fe6aac1c217 compressed=True data_size=30982370 duration=3491067 ended=100002 fail_fast=10 max_runs=100000 pass=100002 priority=100 remaining=0 runtime=0:59:57 sanity=False started=100141 stopped=20220407-061529 submitted=20220407-051532 timeout=5400 username=renxuan
* Revert #6266 .
Remove resolving-related functionalities in connection string. Connection string will be used for storing purpose only, and non-mutable.
20220407-175119-renxuan-55d30ee1a4b42c2f compressed=True data_size=30970443 duration=5437659 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:59:31 sanity=False started=100154 stopped=20220407-185050 submitted=20220407-175119 timeout=5400 username=renxuan
* Add hostname to coordinator interfaces.
* Turn on the new hostname logic.
* Add the corresponding change in config txns.
The most notable change is before calling basicLoadBalance(), we need to call tryInitializeRequestStream() to initialize request streams first.
Passed correctness tests.
* Return error when hostnames cannot be resolved in coordinators command.
* Minor fixes.
2022-04-27 21:54:13 -07:00
79063d5fc4
Better accounting for tenant prefixes when working with key size limits ( #6973 )
...
* Consider tenant prefix sizes when doing key size limit checks
* Fix formatting
2022-04-27 14:38:59 -07:00
cbe4a873d2
Merge remote-tracking branch 'origin/main' into features/validate-trace-events-in-simulation
2022-04-25 17:39:29 -06:00
21336fbf12
Merge remote-tracking branch 'origin/main' into features/authz
2022-04-25 08:45:23 -06:00
Josh Slocum