47a8cebfb3
ddsketch: Make sure that all ctors use 0 < error < 1
2022-11-14 11:41:06 -08:00
7f33b0fa70
clang-format
2022-11-12 14:09:31 -08:00
7ea42841a4
Merge remote-tracking branch 'origin/main' into ddsketch
2022-11-12 13:52:57 -08:00
23706c957b
Use DDSketch for Sample Data.
2022-11-12 13:45:46 -08:00
dcbe51f24e
Merge branch 'main' of github.com:apple/foundationdb into commitproxies
2022-11-11 17:28:01 -08:00
2e4cd67fb8
Add a transaction option to bypass storage quota enforcement.
...
This can be used for transactions that only delete data to allow them to
commit even if the tenant group is over its storage quota.
2022-11-11 16:54:35 -08:00
7e78795284
add bytelimit for prefetch ( #8768 )
...
* add bytelimit for prefetch
A fraction of byteLimit will be used as the limit to fetch index.
For the indexes fetched, fetch records for them in batch.
byteLimit always count the index size, it also count record if exist,
it at least return 1 index-record entry and always include the last entry
despite that adding the last entry despite it might exceed limit.
There is a Knob STRICTLY_ENFORCE_BYTE_LIMIT, when it is set, records
will be discarded once the byteLimit is hit, despite they are fetched.
Otherwise, return the whole batch.
2022-11-11 13:36:06 -08:00
5834517570
Add fdbcli blobrestore to start the full restore
2022-11-11 08:32:23 -08:00
76733ddde6
Merge branch 'main' of github.com:apple/foundationdb into commitproxies
2022-11-09 19:09:06 -08:00
8b489be340
Handle future_version() in tenantaware storage metrics function
2022-11-09 17:50:04 -08:00
105648b888
Merge branch 'main' of github.com:apple/foundationdb into commitproxies
2022-11-09 17:38:30 -08:00
ca572f58fb
Combine *_COST_BYTE_FACTOR knobs into a single TAG_THROTTLING_PAGE_SIZE knob
2022-11-08 07:44:01 -08:00
7dde2a755b
Reject transactions if the tenant is over quota
2022-11-07 15:59:26 -08:00
0adcd92d33
Merge pull request #8461 from hfu94/id
...
Each getRange request in a txn has a distinct trace id
2022-11-07 09:34:22 -08:00
93142e2c29
In try commit, wait for the read version future before waiting for the commit cost estimate. This allows us to give precedence to the database_locked error over errors thrown by the commit cost estimation.
2022-11-02 14:47:24 -07:00
ffbd81cfcf
Merge branch 'main' of github.com:apple/foundationdb into monitorusage
2022-11-01 16:51:27 -07:00
39abc712b0
Merge pull request #8549 from sfc-gh-tclinkenbeard/expose-txn-cost
...
Create `fdb_transaction_get_total_cost` function
2022-11-01 08:14:57 -07:00
6fa5059521
Tweak where TenantInfo is initialized in metrics functions in NativeAPI
...
Also clean up debug trace events
2022-10-31 16:48:06 -07:00
f7b608e53f
blob: refactor blob get tenant code
2022-10-31 10:37:36 -07:00
1ab432e49d
blob: fix error propagation in getBlobRanges()
...
Fixes: 48d6e725c2
2022-10-31 10:37:36 -07:00
0eb1598afa
Merge remote-tracking branch 'origin/main' into expose-txn-cost
2022-10-30 09:36:37 -07:00
82ab299788
Merge branch 'main' of github.com:apple/foundationdb into monitorusage
2022-10-28 15:42:27 -07:00
326d45819e
Merge branch 'main' into storageserver-pml
2022-10-28 14:14:44 -07:00
ba382169d1
Handle `unknown_tenant` error in metrics functions in NativeAPI
2022-10-28 13:37:40 -07:00
0a15f081a1
Proactively clean up idempotency ids for successful commits ( #8578 )
...
* Proactively clean up idempotency ids for successful commits
This change also includes some minor changes from my branch working on
an idempotency ids cleaner, that I'd like to get merged sooner rather
than later.
- Adding a timestamp to idempotency values
- Making IdempotencyId an actor file
- Adding commit_unknown_result_fatal
- Checking idempotencyIdsExpiredVersion in determineCommitStatus
- Some testing QOL changes
* Factor out decodeIdempotencyKey logic
* Fix formatting
* Update flow/include/flow/error_definitions.h
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
* Use KeyBackedObjectProperty for idempotencyIdsExpiredVersion
* Add IDEMPOTENCY_ID_IN_MEMORY_LIFETIME knob
* Rename ExpireIdempotencyKeyValuePairRequest
Also add a code probe for the case where an ExpireIdempotencyIdRequest is
received before the count is known, and add an assert
* Fix formatting and add TODO for nwijetunga
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-10-28 09:07:54 -07:00
d53ed6acae
Merge branch 'main' into storageserver-pml
2022-10-28 00:14:01 -07:00
dc60f63f9b
Revert "Cancel watch when the key is not being waited"
...
This reverts commit 639afbe62c
2022-10-27 19:46:05 -07:00
634bd529e7
Revert "Record the version of each watch"
...
This reverts commit 4bd24e4d64
2022-10-27 19:46:05 -07:00
e460933b52
Revert "Remove debugging output"
...
This reverts commit 41d1d6404d
2022-10-27 19:46:05 -07:00
e7fd3eda00
Revert "Update fdbclient/NativeAPI.actor.cpp"
...
This reverts commit 812243bafa
2022-10-27 19:46:05 -07:00
f9ad7fb35b
Merge origin/main into storageserver-pml
2022-10-27 18:00:11 -07:00
812243bafa
Update fdbclient/NativeAPI.actor.cpp
...
Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>
2022-10-27 12:42:05 -07:00
41d1d6404d
Remove debugging output
2022-10-27 12:42:05 -07:00
4bd24e4d64
Record the version of each watch
...
In the case
1. A watch to key A is set, the watchValueMap ACTOR, noted as X, starts waiting.
2. All watches are cleared due to connection string change.
3. The watch to key A is restarted with watchValueMap ACTOR Y.
4. X receives the cancel exception, and tries to dereference the counter. This causes Y gets cancelled.
the reference count will cause watch prematurely terminate. Recording
the versions of each watch would help preventing this issue
2022-10-27 12:42:05 -07:00
639afbe62c
Cancel watch when the key is not being waited
...
Currently, there is a cyclic reference situation in
DatabaseContext -> WatchMetadata -> watchStorageServerResp ->
DatabaseContext
If there is a watch created in the DatabaseContext, even the
corresponding wait ACTOR is cancelled, the WatchMetadata will still hold
a reference to watchStorageServerResp ACTOR, which holds a reference to
DatabaseContext.
In this situation, any DatabaseContext who held a watch will not be
automatically destructed since its reference count will never reduce to
0 until the watch value is changed. Every time the cluster recoveries,
several watches are created, and when the cluster restarts, the
DatabaseContext which not being used, will not be able to destructed due
to these watches.
With this patch, each wait to the watch will be counted. Either the
watch is triggered or cancelled, the corresponding count will be
reduced. If a watch is not being waited, the watch will be cancelled,
effectively reduce the reference count of DatabaseContext. This will
hopefully fix the issue mentioned above.
The code is tested by 1) Manually change the number of logs of a local
cluster, see the cluster recovery and previous DatabaseContext being
destructed; 2) 100K joshua run, with 1 failure, the same test will fail
on the current git main branch.
2022-10-27 12:42:05 -07:00
56abec32f1
Bug fix: The change feed request UID is actually not just for debugging and can't be shared across requests, so the debugID in ReadOptions should not be used. Restored the original ChangeFeedRequest member but renamed it from debugUID to just id.
2022-10-26 20:45:39 -07:00
fb44945a89
Use id variable to simplify logic a bit.
2022-10-26 17:35:39 -07:00
deeedfc3f8
Merge pull request #8537 from sfc-gh-dzhou/unblob
...
blob: allow purge ranges to begin and end in unblobbified regions
2022-10-26 11:11:09 -07:00
c2cacb4123
Added ReadOptions and read lock support support to ChangeFeed requests and used its DebugID to replace the streamUID. Refactored read lock and read options usage in Storage Server to simplify code and handle ReadOptions to Priority conversion in a single place, fixed a few bugs along the way. Set CacheResult to False for requests created by fetchKeys().
2022-10-26 01:32:45 -07:00
bb0236433c
Merge pull request #8540 from sfc-gh-xwang/feature/main/storageMetrics
...
Make MockStorageServer serve StorageMetrics related request
2022-10-25 17:29:21 -07:00
36d9de9072
change UNREACHABLE to ASSERT(false); change function name
2022-10-25 15:43:24 -07:00
474955b10d
blob: allow purge as long as it doesn't start in a range
2022-10-24 11:08:04 -07:00
48d6e725c2
blob: convert listBlobbifiedRangesActor() to take a Transaction
2022-10-24 11:08:04 -07:00
07f1247885
blob: getBlobRanges() use transaction instead of RYWTransaction
2022-10-24 11:08:04 -07:00
dd52d997ba
blob: rename rowLimit variable for listBlobbifiedRanges()
2022-10-24 11:08:04 -07:00
e07a50573a
splitStorageMetrics finish implementation (no unit test yet but 100k
...
test pass)
2022-10-24 09:58:41 -07:00
1603926595
refactoring old waitStorageMetrics and finish MGS::waitStorageMetrics
...
(no unit test yet)
2022-10-24 09:58:41 -07:00
70eb9aef28
simple MGS IKeyLocationService implementation
2022-10-24 09:58:41 -07:00
32ae7bb529
Merge remote-tracking branch 'origin/main' into expose-txn-cost
2022-10-23 12:59:07 -07:00
b442705dc7
Change units for tag quota enforcement from pages to bytes
2022-10-23 12:57:19 -07:00
da4ceaf735
Improve code coverage for proxy-level transaction tag throttling
2022-10-23 10:39:59 -07:00
1ae98808f9
Retry on proxy_tag_throttled errors
2022-10-23 09:56:30 -07:00
e5a5ec36a4
Merge commit '0872cbfb2f00886817f18584d95af217e28ad51d' into storageserver-pml
...
# Conflicts:
# fdbserver/storageserver.actor.cpp
2022-10-19 13:25:31 -07:00
92bbcebed9
Increase trState->totalCost by one with each clear
2022-10-18 14:05:46 -07:00
89519343a7
adding new bg read metrics to client and refactoring them into a new trace event ( #8493 )
2022-10-18 14:13:35 -05:00
1b2fcdd4f6
Merge pull request #8053 from sfc-gh-akejriwal/getsizetenant
...
Make the storage metrics functions tenant aware
2022-10-17 20:02:52 -07:00
6251e8be32
Track transaction cost for range reads
2022-10-17 10:08:49 -07:00
1c90fbbc61
blob: tenant verifyBlobRange() native api
2022-10-17 11:42:08 -05:00
4a33de31e9
blob: tenant listBlobbifiedRanges() native api
2022-10-17 11:41:45 -05:00
120b38a926
blob: tenant (un)blobbify() native api
2022-10-17 11:41:12 -05:00
0ee0ce62f0
blob: blobGranuleGetTenantEntry() for tenant and database calls
...
Currently, blobGranuleGetTenantEntry() expects to get the TenantName
through the transaction. Database calls cannot abide by this contract.
So enable the call to take in a TenantName and expect at least 1 to be
set.
2022-10-17 11:41:12 -05:00
84aa815026
Implemented first version of Transaction::getTotalCost, along with workload
2022-10-16 21:58:08 -07:00
0f9da9d1ad
Merge branch 'main' of github.com:apple/foundationdb into getsizetenant
2022-10-14 16:50:51 -07:00
854212fe94
Incorportate code review suggestions
2022-10-13 17:41:31 -07:00
be64b12ecf
Clean up TraceEvents, TODOs, and simplify workload.
2022-10-13 14:31:42 -07:00
df5825ff65
Merge pull request #8398 from sfc-gh-anoyes/anoyes/idempotency-id2
...
Initial work for automatic idempotency
2022-10-13 13:07:14 -07:00
293cc2a8fb
Each getRange request in a txn has a distinct trace id
...
Previously when multiple getRange requests are from the same parent
RL request, they share the same trace id from TransactionState, with
this change it has better granularity and makes it easier to compare
latency between getRange requests.
2022-10-12 12:12:58 -07:00
96574bacaa
Change Feed TSS Support ( #8384 )
...
* Change Feed TSS Support
* bug fixing for rare mismatch cases
* Adding rollback handling to tss change feed comparison
2022-10-12 10:23:51 -05:00
3223208f0c
Explicitly pass `arena` to the `withPrefix` function.
...
Also update some TraceEvents to be more succinct.
2022-10-11 18:35:01 -07:00
aad3899cb7
Enforce the lifetime of a write tx is actually <= MAX_WRITE_TRANSACTION_LIFE_VERSIONS
2022-10-11 13:46:39 -07:00
045452aec8
Add more detail to TraceEvents
2022-10-11 13:46:39 -07:00
6ef8d7ec95
Add AutomaticIdempotencyWorkload
2022-10-11 13:46:39 -07:00
f48d0d929b
Assign to idempotency id through transaction options
2022-10-11 13:46:39 -07:00
11658c7135
Merge branch 'main' of github.com:apple/foundationdb into getsizetenant
2022-10-11 13:09:47 -07:00
12aaabb283
Fix bugs in finding correct shards for tenant aware storage metrics.
...
* Account for tenant in during cache invalidation in NativeAPI actor.
* Account for tenant prefix while checking whether a shard is readable
on storage server.
This commit also adds many debug TraceEvents that will later be
removed.
2022-10-10 18:52:03 -07:00
b7ce834d28
Merge commit '7c89cd705faee52d5d78e6c77665cb7cc4502f58' into storageserver-pml
2022-10-07 11:39:42 -07:00
ea1325a552
Merge pull request #8319 from sfc-gh-tclinkenbeard/add-rare-code-probe-annotation
...
Add `rare` code probe decoration
2022-10-07 09:39:00 -06:00
3228afefd3
Unrevert #7578 - storage server PriorityMultiLock and PML rewrite.
2022-10-06 23:41:28 -07:00
f8186988bc
Merge pull request #8378 from sfc-gh-ajbeamon/tenant-support-for-database-switching
...
Add tenant support for database switching
2022-10-05 14:57:33 -07:00
e83229ed8b
- Version vector specific: Propagate the latest commit version, as ( #8393 )
...
part of the read request, when reading from a storage server.
2022-10-05 16:23:56 -04:00
dc917453c1
Targeted blob granules fault injection ( #8231 )
2022-10-05 13:44:38 -05:00
53593c77ac
When updating watches after a database switch, lookup the tenant again to get the ID on the new cluster. Update the switching test to set up tenants properly.
2022-10-03 09:24:13 -07:00
de90e1aab1
Improve the code in the actors and server, simplify the workload
2022-09-29 18:34:35 -07:00
f63934117d
Make the storage metrics function tenant aware [WIP]
...
This change makes the getEstimatedRangeSizeBytes function tenant aware.
Previously, this function would return the size of the requested
keyspace even if the tenant in the Transaction or DatabaseContext did
not match the tenant corresponding to the keyspace.
Also make some improvements to the new workload.
2022-09-29 18:32:04 -07:00
c8bb15e8ee
Merge pull request #8216 from sfc-gh-ajbeamon/backup-support-for-tenants
...
Include tenant and metacluster metadata in backups
2022-09-28 12:00:24 -07:00
985958c260
Add rare code probe decoration
2022-09-25 15:28:32 -07:00
5774249e5b
Revert "[DRAFT] Redwood PriorityMultiLock enable different launch limits to be specified based on different priority level."
2022-09-23 12:22:47 -06:00
f78eb8c778
Adding bg read amp metrics ( #8275 )
2022-09-22 16:31:27 -07:00
fda0d7223d
Update backup to include system key ranges needed for tenants. Run simulated backup tests with tenants.
2022-09-22 10:00:13 -07:00
04b4960786
Merge branch 'main' into fzhao/RedwoodIOLaunchLimit
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/ReadWrite.actor.cpp
2022-09-22 00:39:51 -07:00
7a09b701cc
Merge pull request #8141 from sfc-gh-jfu/network-disable-bypass
...
Introduce network option for disabling mvc bypass
2022-09-21 17:33:48 -07:00
4ea4546cb6
blob/java: verifyBlobRange() with latestVersion
2022-09-21 14:07:16 -07:00
e353169a50
blob: teach purge about latestVersion
...
This teaches purgeBlobGranules about latestVersion and rejects
versions <= 0.
2022-09-21 14:04:58 -07:00
4e8ac6c811
remove network option on native side since option is no longer being forwarded
2022-09-20 11:10:12 -07:00
1f778f9d76
Merge branch 'main' of github.com:apple/foundationdb into network-disable-bypass
2022-09-20 09:30:53 -07:00
0f3f493c28
Merge pull request #8218 from sfc-gh-jslocum/aligned_purge
...
fixes for non-aligned blob range calls
2022-09-19 15:00:02 -05:00
4fd64630e8
Convert literal string ref instances to use _sr suffix
2022-09-19 11:35:58 -07:00
88f88707f5
fixes for non-aligned blob range calls
2022-09-16 19:06:15 -05:00
1abac8ea9f
check on shared state ptr in native api and add to test spec in api tester
2022-09-16 15:11:33 -07:00
7b4598a53d
Merge pull request #8197 from sfc-gh-jslocum/bg_code_coverage_cleanup
...
cleaning tss and blob granule file code probes
2022-09-16 09:32:20 -07:00
Kevin Hoxha