0ab691b707
Merge pull request #10002 from sfc-gh-jazhuang/readThrough
...
Fix RangeResult.readThrough misuse
2023-04-27 09:59:11 -07:00
a05e078c4a
Remove locations.size() == expectedShardCount assertion and add comments
2023-04-26 14:23:09 -07:00
7f6d5f296a
Merge commit 'e318fc260070ba6ba604930b8f259c9b655938ea' into keybackedrangemap
...
# Conflicts:
# flow/include/flow/error_definitions.h
2023-04-25 14:21:23 -07:00
6b15d67928
Merge pull request #10010 from jzhou77/main
...
Properly handle proxy_memory_limit_exceeded error for GetKeyServerLocationsRequest
2023-04-25 11:18:03 -07:00
74bb659f71
Simplify backoff calls per comment
2023-04-25 09:15:16 -07:00
c544985fe5
Add trace events and adjust backoff
...
For each success, half the backoff until less than initial backoff value, then
set the backoff to 0.
2023-04-20 15:56:06 -07:00
a83295e3bd
Add backoff to lookupTenantImpl for commit_proxy_memory_limit_exceeded error
2023-04-19 16:55:46 -07:00
3bfd353a22
Add backoff to getKeyLocation_internal as well
2023-04-19 16:45:50 -07:00
ad1c7bba74
Make commit_proxy_memory_limit_exceeded error a database level backoff
...
Since this error means the database is overloaded.
2023-04-19 15:52:16 -07:00
b49625d45b
Properly handle proxy_memory_limit_exceeded error for GetKeyServerLocationsRequest
...
Also add buggify to inject the error in simulation.
2023-04-19 09:35:09 -07:00
53ee26d758
Changed KeyBackedTypes to an actor file. Added TypedKeySelectors for Map and Set classes and getRange() keySelector methods. Added debug macro for KeyBackedTypes. Rewrote KeyBackedRangeMap using keyselectors on KeyBackedMap.
2023-04-18 22:21:19 -07:00
6b8e35fd19
Merge pull request #9964 from sfc-gh-xwang/fix/main/comments
...
fix HealthMetrics update bug and correct comments about readLoadKSecond()
2023-04-18 16:42:42 -07:00
09ab501cb7
Improve logging for test ( #9966 )
...
In clearData() used by simulation test to clean up, we re-use the same debugID for multiple transactions,
making it less clear when grepping for the debugID. This PR assigns a new debugID for each new transaction.
Some of the commented-out tracing code are already obsolete, I updated several of them, but am not sure if
the tracing should be enabled when enabling debug transaction.
Finally, use different but similar with the same prefix error messages for different call stacks.
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-04-18 15:25:06 -07:00
b7da2ed16c
Fix RangeResult.readThrough misuse
...
Fix `RangeResult.readThrough` misuses:
1. KeyValueStores do not need to set readThrough, as it will not be
serialized and return. Also setting it to the last key of the result
is not right, it should at least be the keyAfter of the last key;
2. Fix NativeAPI doesn't set `RangeResult.more` in a few places;
3. Avoid `tryGetRange()` setting `readThrough` when `more` is false,
which was a workaround for the above item 2;
4. `tryGetRangeFromBlob()` doesn't set `more` but set `readThrough` to
indicate it is end, which was following the same above workaround I
think. Fixed that.
5. `getRangeStream()` is going to set `more` to true and then let the
`readThrough` be it's boundary.
Also added readThrough getter/setter function to validate it's usage.
2023-04-17 21:37:51 -07:00
bdab07cbc8
fix detailed HealthMetrics update bug
...
add omitted return value
2023-04-17 12:24:14 -07:00
29161b2fda
Revert matchIndex feature
...
It is not protocol compatible, revert it to avoid deployment issue.
Will have a new PR to have the feature if moving forward.
2023-04-17 09:39:45 -07:00
370feaa3c9
refactoring and adding future compatibility to blob range metadata ( #9955 )
...
* refactoring and adding future compatibility to blob range metadata
* formatting
2023-04-13 15:06:50 -05:00
67b737b44d
add getStorageStats method to DatabaseContext
2023-04-12 09:33:05 -07:00
d903d0cc8d
checkSafeExclusion should always create new ExclusionSafetyCheckRequest
2023-03-31 16:27:26 -07:00
64b6a5d257
Allow boolean parameters to be nested inside of namespaces or classes
2023-03-30 15:09:59 -07:00
894f555e95
Test loop profiler using API Tester ( #7174 )
...
* Api Tester: Specify knobs in the toml file; Test loop profiler
* Gracefully stop the loop profiler thread
* Protect loop profiler thread by mutex
* Create loop profiler thread only if is not stopped
2023-03-30 23:00:23 +02:00
c7fcf1e8f2
handling multiple in-flight rollbacks for tss change feed stream comparison ( #9816 )
2023-03-28 09:39:38 -05:00
81c3cb8c50
Psm checkpoint ( #9636 )
...
* Update NativeAPI getCheckpointForRange().
* Implemented checkpoint in SS.
* clean up.
* Disabled StorageServerCheckpointTest.
* Serialized checkpoint creation and deletion.
Simplified checkpoint GC, via deleting CheckpointMetaData::dir.
* Fixed PhysicalShardMove test. Where fetchCheckpoint target range is misset.
* Minor improvements on CheckpointMetaData and DataMoveMetaData.
* fmt.
* Optimized PhysicalShardMove test
cleanup.
* dismiss operation_obsolete, and throw actor_cancelled.
* fmt.
* Resolved commments.
2023-03-21 09:14:10 -07:00
216d0be2cf
Add processID, networkAddress, and locality to layer status JSON for Backup Agents. ( #9736 )
...
* Add processID, networkAddress, and locality to layer status JSON for Backup Agents.
* Backup/dr agent determines network address to report in Layer Status only once, when the status updater loop begins, since it is a blocking call which connects to the cluster. And lots of code cleanup.
2023-03-17 18:07:03 -07:00
52c0dc56cc
fixing 2 bugs related to high delta file waitCommitted latency
2023-03-15 08:39:42 -05:00
c43f8b3fdc
Refactor - introduce BlobRestoreController for APIs to manage restore state ( #9616 )
2023-03-08 07:50:30 -08:00
7273723a43
Add the hotrange fdbcli command ( #9570 )
...
* Add the hotrange fdbcli command
* Remove the unnecessary state
* Add the doc about the hotrange command
2023-03-06 14:46:52 -08:00
40ed365303
fixing checkBlobSubRange to not increase version every retry
2023-03-02 09:34:55 -06:00
a350a929b9
Merge pull request #9494 from sfc-gh-jslocum/bg_cp_improvements
...
addressing review comments and fixmes in bg commit proxy code
2023-02-28 12:30:58 -08:00
5bebb5b4aa
Merge pull request #9492 from sfc-gh-vgasiunas/vgasiunas-api-version-defs
...
Centralize definition of API Version for Java, Python and C API
2023-02-28 12:04:02 -07:00
469e77158f
Add metacluster support for tenant locking
2023-02-27 16:53:13 -08:00
716a9c3817
addressing review comments and fixmes in bg commit proxy code
2023-02-27 10:51:47 -06:00
ba726fac87
Replace hardcoded API version checks for 720 and 730
2023-02-27 16:18:01 +01:00
6187811f71
Reworking getBlobGranuleRanges to also use commit proxy rpc for authz, and adding test ( #9470 )
2023-02-24 17:15:32 -06:00
8695fc15fc
Merge remote-tracking branch 'origin/main' into features/tenant-lock2
2023-02-22 13:12:23 -07:00
23df46773d
Merge pull request #9422 from sfc-gh-satherton/client-read-options
...
Add transaction option definitions for read priority and read cache
2023-02-22 09:00:25 -08:00
bf97c3dbce
adding java tenant blob management test and fixing bug it found ( #9428 )
2023-02-22 10:52:26 -06:00
a21b2fe9f9
Simplified read priority option to three separate options for normal/low/high priority.
2023-02-21 22:48:38 -08:00
5969616af8
Merge commit '6de85e7cd8e9dd74a571de9e04679e669bcbb5b6' into client-read-options
2023-02-21 20:46:20 -08:00
15d8548c0e
Merge remote-tracking branch 'origin/main' into features/tenant-lock2
...
# Conflicts:
# fdbserver/ApplyMetadataMutation.cpp
# fdbserver/storageserver.actor.cpp
2023-02-21 13:39:35 -07:00
958f3b531b
Plumbing blob worker mapping through commit proxy like storage server ( #9401 )
...
* Plumbing blob worker mapping through commit proxy like storage server mapping
* review comments
* formatting
2023-02-21 13:21:44 -06:00
246fd1dd4e
Remove auto cache option since there is no meaningful implementation of this yet. Change places using trState in a native Transaction to set cache mode or Low/Normal/High priority to use the new transaction options instead.
2023-02-21 02:50:30 -08:00
9bf28899d4
Add transaction option definitions for read priority and read server side cache mode.
2023-02-21 00:54:35 -08:00
398079db3a
Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration
2023-02-20 17:54:06 -08:00
bfb3ffc509
added c and java apis for granule flush ( #9412 )
2023-02-20 10:28:11 -06:00
1aef6cb5f7
Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration
2023-02-17 20:41:59 -08:00
6c2fb13173
adding wait parameter to blobbify api ( #9360 )
...
* adding wait parameter to blobbify api
* formatting
* fixing comment style
* fixing bug and adding debugging
* adding blob ranges unit test
* testing both blobbify cases in cancel
* formatting
* switch to explicit blocking api instead of boolean flag
* remove comments
* format
2023-02-17 12:20:53 -06:00
d9c126a2d9
Introduce WipedString for Arena block holding AuthZ tokens ( #9381 )
...
* Enable secure allocation mode in Arena
This mode allows zeroing out blocks holding sensitive data after use
* Introduce WipedString to all token-holding memory
Also introduce a option flag "sensitive"
* Make pointer equivalency a hard requirement for non-ASAN builds
So that we can detect when Arena/malloc/memory-wipe behavior changes
2023-02-16 10:44:32 +01:00
3017e15448
Merge remote-tracking branch 'origin/main' into features/tenant-lock2
...
# Conflicts:
# fdbclient/include/fdbclient/TenantManagement.actor.h
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/TenantManagementWorkload.actor.cpp
2023-02-13 11:43:07 +01:00
1e5bac6238
fixing fault injection stalling change feed fetch ( #9316 )
2023-02-08 15:49:56 -06:00
Jay Zhuang