508429f30d
Redwood chunked file growth and low priority IO starvation prevention ( #5936 )
...
* Redwood files now growth in large page chunks controlled by a knob to reduce truncate() calls for expansion. PriorityMultiLock has limit on consecutive same-priority lock release. Increased Redwood max priority level to 3 for more separation at higher BTree levels.
* Simulation fix, don't mark certain IO timeout errors as injected unless the simulated process has been set to have an unreliable disk.
* Pager writes now truncate gradually upward, one chunk at a time, in response to writes, which wait on only the necessary truncate operations. Increased buggified chunk size because truncate can be very slow in simulation.
* In simulation, ioTimeoutError() and ioDegradedOrTimeoutError() will wait until at least the target timeout interval past the point when simulation is sped up.
* PriorityMultiLock::toString() prints more info and is now public.
* Added queued time to PriorityMultiLock.
* Bug fix to handle when speedUpSimulation changes later than the configured time.
* Refactored mutation application in leaf nodes to do fewer comparisons and do in place value updates if the new value is the same size as the old value.
* Renamed updatingInPlace to updatingDeltaTree for clarity. Inlined switchToLinearMerge() since it is only used in one place.
* Updated extendToCover to be more clear by passing in the old extension future as a parameter. Fixed initialization warning.
2021-11-12 13:47:07 -08:00
4979ccb889
commits recovered if written to every tlog minus failure tolerance.
2021-11-12 12:10:04 -05:00
ad9ff18f69
Flip PROXY_USE_RESOLVER_PRIVATE_MUTATIONS to true
2021-11-11 11:26:18 -08:00
394b9dc619
Code review changes
2021-11-10 11:53:27 -05:00
f6342b0a8d
Update defaults
2021-11-10 11:51:05 -05:00
66520eb1c1
Utilize read types to do selective throttling
2021-11-10 11:51:04 -05:00
fdb3b72e35
Introduce GetRangeAndFlatMap to push computations down to FDB
...
Re-introduce #5609
2021-11-09 13:52:28 -08:00
586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB"
2021-11-04 08:46:56 -07:00
0853661d13
Introduce getRangeAndHop to push computations down to FDB
2021-11-03 13:21:16 -07:00
befe1993c4
fix conflict on rebase
2021-10-29 12:25:26 -04:00
7b0c39572e
- Disable PROXY_USE_RESOLVER_PRIVATE_MUTATIONS in simulation tests
2021-10-28 21:31:53 +00:00
1a2a838df3
add knob
2021-10-27 09:08:37 -07:00
69190ed04e
format
2021-10-27 09:08:37 -07:00
0053b4793e
change knob and delete redundant doBuildTeam
2021-10-27 09:08:37 -07:00
2208b04174
Merge pull request #5855 from sfc-gh-etschannen/blob_full_clean
...
Blob Granules V0
2021-10-26 09:57:35 -07:00
c96f560cbe
Verify rollback of a single version in simulation, other small fixes
2021-10-25 12:03:22 -07:00
0ff8ddc2b6
Merge branch 'master' into blob_full_clean
2021-10-25 13:38:48 -05:00
d153519188
Merge pull request #5813 from sfc-gh-jslocum/ss_ebrake_streaming_fix
...
Fixes to ss e-brake, tlog streaming, and their interaction
2021-10-22 10:46:17 -07:00
773886515e
Merge branch 'feature-range-feed' into blob_full_clean
2021-10-22 11:07:51 -05:00
0cf829ef91
Reduce restore error message
2021-10-20 14:02:48 -07:00
48a0ecd647
Merge pull request #5787 from dlambrig/integrate-PR5700
...
version vector / Calculate TPCV on resolvers
2021-10-20 16:35:44 -04:00
8dd7f8f447
Fixes to ss e-brake, tlog streaming, and their interaction
2021-10-20 10:48:29 -05:00
ff0d687704
Cleanup comments for server knobs.
2021-10-18 17:45:30 -04:00
246f035afe
Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
2021-10-18 19:07:33 +00:00
eb814ce070
Remove dead code, check replies match for all resolvers
2021-10-18 10:23:08 -04:00
507a09893c
Add ClientCount to ClusterControllerMetrics ( #5748 )
2021-10-17 20:47:11 -07:00
eaf0a00502
Enable PROXY_USE_RESOLVER_PRIVATE_MUTATIONS for simulation tests
2021-10-15 09:47:23 -04:00
0dc9c607f4
Add knob PROXY_USE_RESOLVER_PRIVATE_MUTATIONS
...
To control proxy to use private mutations from resolvers or not.
2021-10-15 09:47:23 -04:00
5f0ec0612a
Merge branch 'feature-range-feed' into blob_full
2021-10-13 15:44:35 -05:00
2ec8781224
Merge knobs into one.
2021-10-13 14:00:37 -04:00
a163619fbc
Change default val for knob.
2021-10-13 09:58:09 -04:00
5a6a052c55
Add a knob to gate blob-related work.
2021-10-13 09:48:02 -04:00
fdb67a2abf
Merge pull request #5754 from dlambrig/always-true
...
have ENABLE_VERSION_VECTOR true by default
2021-10-11 18:41:04 -04:00
ca8f4dd9d0
ENABLE_VERSION_VECTOR true
2021-10-11 18:38:54 -04:00
f68d018a06
conflicts
2021-10-08 12:39:39 -04:00
c0fbe5471f
Implement the core logic of grey failure triggered failover
2021-10-07 11:19:34 -07:00
4d54669ccd
Recruit the blob workers via blob manager.
...
In this PR, the blob manager now recruits blob workers
(via communication with the cluster controller). Blob workers
are onboarded as blob worker processes enter the cluster.
2021-10-04 11:07:08 -04:00
58e1888d8e
remove network hop by getting previous commit versions in GetCommitVersionRequest
2021-09-30 11:51:57 -04:00
cb154c12d4
disable ENABLE_DETAILED_TLOG_POP_TRACE by default
2021-09-26 21:46:57 -07:00
c2885ab70d
The BACKUP_LOCK_BYTES knob could be buggified to a value that was too low, resulting in backup getting stuck.
2021-09-24 09:15:30 -07:00
5fa6c687d6
Add blob manager as a singleton.
2021-09-23 10:45:37 -04:00
72edcd8d73
Address PR comments.
...
Revert knob name change, fix comparison between new and old
recruitments, and get rid of empty `if` block.
2021-09-22 16:56:34 -05:00
0b6fecddbc
Refactor logic for recruiting singletons.
...
This commit refactors the logic for recruiting singletons,
which is done by the ClusterController. This allows for far
easier additions of new singletons in the future, and also
cleans up the code.
Also, the logic for recruiting DD was changed to mirror
the logic for recruiting RK. Although the logic for RK
allows there to be many RKs existing at once, the moveKeysLock
mechanism used by DD still prevents multiple DDs existing at once.
2021-09-22 16:56:18 -05:00
6533678f0d
Address PR comments.
...
Revert knob name change, fix comparison between new and old
recruitments, and get rid of empty `if` block.
2021-09-20 14:26:42 -05:00
fe098b3b11
Refactor logic for recruiting singletons.
...
This commit refactors the logic for recruiting singletons,
which is done by the ClusterController. This allows for far
easier additions of new singletons in the future, and also
cleans up the code.
Also, the logic for recruiting DD was changed to mirror
the logic for recruiting RK. Although the logic for RK
allows there to be many RKs existing at once, the moveKeysLock
mechanism used by DD still prevents multiple DDs existing at once.
2021-09-20 14:26:42 -05:00
c9b8bdffbe
Passes a single correctness test!
2021-09-15 17:18:04 -05:00
c2d1d1704f
Merge branch 'feature-range-feed' into blob_full
2021-09-10 11:21:52 -05:00
eb76343dfb
Added blob granule reassignment and splitting
2021-09-08 14:09:14 -05:00
8689e1f106
merge with master
2021-08-30 15:29:08 -04:00
be440ab954
Merge pull request #5260 from FuhengZhao/RedwoodHistogram
...
Redwood local histograms
2021-08-26 12:05:44 -07:00
8d49c98a41
Added simulation workload for blob granules and fixed some bugs
2021-08-26 13:48:05 -05:00
e328e2b5df
format fix
2021-08-24 18:34:44 -07:00
5eb92ae65c
reomve registry
2021-08-24 15:47:04 -07:00
5259af787d
Switched blob implementation to use backup container
2021-08-24 13:47:47 -05:00
57652a0cae
Format
2021-08-24 10:00:24 -07:00
b65a66fab7
log redwood histogram seperatly
2021-08-24 09:57:39 -07:00
d554205db3
Move common knob initialization macros to header
2021-08-23 11:41:01 -07:00
7998ec4232
Add atomicity to knobs
2021-08-23 11:41:00 -07:00
02b3ed3ff1
Adding deadline option to rocksdb calls.
2021-08-19 14:11:28 -07:00
fbb393f998
Added readrange timeoout check and rocksdb read deadline option.
2021-08-19 14:11:28 -07:00
24ac173c95
Adding ReadRangeAction timeout, returning error and using timer_monotonic changes.
2021-08-19 14:11:28 -07:00
01e85610ab
Cancelling the timedout reads with rocksdb storage.
2021-08-19 14:11:28 -07:00
45b40addb7
Merge pull request #5379 from neethuhaneesha/eagerReadsDisable
...
Disabling option for removing eagerReads for ClearRange mutations.
2021-08-19 12:39:30 -04:00
c45daf6f51
Disabling option for removing eagerReads for ClearRange mutations.
2021-08-13 01:26:50 -07:00
df7a801945
remove FIXME
2021-08-12 14:10:34 -07:00
a97570bd06
solve mis-spelling, trace log and format problems
2021-08-11 18:26:00 -07:00
921a2cfca1
Merge branch 'feature-range-feed' into blob_full
2021-08-10 11:25:48 -05:00
247602753d
Blob worker handles multiple ranges now
2021-08-10 11:19:43 -05:00
e83b27387a
POC worker that dumps fdb range to an s3 object
2021-08-06 11:01:37 -05:00
1758c92683
- Pull changes related to tlog-peeks from the version indexer branch
...
Pull commits 5e37bc37a095e85aaffb
2021-08-06 14:42:35 +00:00
a081c0baa5
Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
2021-08-05 22:40:32 +00:00
fd74a16f35
format code
2021-08-02 14:24:20 -07:00
2df0474fec
merge master
2021-08-02 11:58:35 -07:00
a26dbba66f
Add RocksDB metrics
2021-07-30 13:27:43 -04:00
10c82b422f
merge master branch
2021-07-28 14:19:46 -07:00
55094bdd00
Merge branch 'master' into expired-forward
2021-07-26 19:58:29 -04:00
48d4ea6d57
Merge pull request #5231 from sfc-gh-tclinkenbeard/availability-improvements
...
Throw `failed_to_progress` error when a role fails to make progress
2021-07-26 15:55:14 -07:00
e39cfd48c3
Ignore goodRecruitmentTime and populate default PEER_LATENCY_CHECK_MIN_POPULATION
2021-07-26 10:24:26 -07:00
34fecb61de
Reject connections to clusters forwarded in the (configurable) past
2021-07-26 10:18:10 -04:00
1057835e8b
merge with master
2021-07-20 17:09:34 -07:00
6836e49073
Throw error when commitProxy gets stuck
2021-07-19 21:00:28 -07:00
d46feb5468
Add COORDINATOR_LEADER_CONNECTION_TIMEOUT server knob
2021-07-19 20:17:46 -07:00
0f99d449fc
Revert "Merge pull request #5194 from apple/revert-5144-rocksdb-in-simulation"
...
This reverts commit 8ecff8c730c8e18f99f0
2021-07-16 12:01:01 -04:00
b7b845dab8
Revert "RocksDB in simulation"
2021-07-15 16:41:33 -07:00
1584ed5853
Merge branch 'master' of https://github.com/apple/foundationdb into tlog_dev
2021-07-14 16:20:19 +00:00
4e16e2552b
Even smaller memtable
2021-07-13 12:25:11 -04:00
0613ff892f
Enable RocksDB in simulation
2021-07-13 12:25:11 -04:00
d8122ebc09
Respond to review comments 7/8
2021-07-12 12:13:25 -04:00
a107dd655e
tpcv messaging
2021-07-12 11:46:52 -04:00
501dc339a9
relax perpetual wiggle pause condition; add trace log; correct perpetual wiggle priority setting
2021-07-12 05:46:55 +00:00
2abdbff11f
add Knobs
2021-07-12 05:33:23 +00:00
8773cb1a42
Merge pull request #5088 from halfprice/zhewu/cc-health-triggered-recovery
...
Implement the main logic of gray network triggered recovery in Cluster Controller
2021-07-11 20:50:34 -07:00
020371a78f
Merge remote-tracking branch 'origin/master' into add-boolean-param
2021-07-07 16:50:51 -07:00
cc10c9aee2
clean up add_trace_event_to_tLog_pop
2021-07-07 14:14:59 -05:00
9b969b99f4
Merge pull request #5118 from sfc-gh-satherton/redwood-rangeread-prefetch
...
Redwood range read prefetching
2021-07-07 11:12:18 -07:00
b82a3f4276
add trace event to tLog pop
2021-07-07 12:13:49 -05:00
569de5259f
Merge branch 'master' of https://github.com/apple/foundationdb into tlog_dev
2021-07-06 23:15:20 +00:00
e74f5d45b0
Added range read prefetching back to Redwood with a conservative approach that should rarely read blocks not needed for the range read. Removed unused knob declarations.
2021-07-04 01:51:44 -07:00
8cc40e3a2b
Expand use of BOOLEAN_PARAM
2021-07-02 21:41:50 -07:00
b50fda6b4b
add simple streaming peek functions
2021-07-01 23:17:28 +00:00
5a325aded9
Implement the core logic of gray network triggered recovery in cluster controller
2021-06-30 11:27:05 -07:00
865367fb72
Add updateWorkerHealth interface in cluster controller
2021-06-24 19:42:28 -07:00
7b713f7fd2
add knob
2021-06-23 05:49:55 +00:00
244e8f3527
Merge branch 'master' of https://github.com/apple/foundationdb into feature-streaming-reply-master
...
# Conflicts:
# fdbclient/CMakeLists.txt
# fdbclient/Knobs.h
# fdbserver/Knobs.h
2021-06-21 16:35:21 -07:00
6619545c30
Merge remote-tracking branch 'origin/master' into config-db
2021-06-18 19:59:56 -07:00
41c790b299
Merge remote-tracking branch 'origin/master' into config-db
2021-06-10 22:31:23 -07:00
83a0e473e8
Refactor IKnobCollection code
2021-06-09 20:50:00 -07:00
a775f92fca
Merge remote-tracking branch 'origin/master' into config-db
2021-06-01 15:39:34 -07:00
ca0893571c
Move server knobs into fdbclient
2021-06-01 03:12:47 -07:00
Steve Atherton