283fd3af27
Add a knob which controls writing prefix compressed kvs mem snapshots.
...
Which will be set to on by default in 7.0
2020-05-12 17:01:52 -07:00
a8e0f1d581
removed knob
2020-05-11 18:20:46 -07:00
d0b414ddf2
Merge pull request #3121 from etschannen/master
...
Added a large random delay on failure detection
2020-05-10 17:59:51 -07:00
b1bd5ef83e
Merge pull request #3120 from satherton/feature-redwood
...
Redwood read concurrency limit, some knobs, and memory-only Pager mode.
2020-05-10 17:34:41 -07:00
07111f0e41
add a large random delay on failure detection so that not all storage servers need to attempt to become the cluster controller
2020-05-10 17:09:33 -07:00
43f9e4dfad
Implemented concurrent read limit in IKeyValueStore interface for Redwood. Added knobs for Redwood page size, concurrent read limit, and page fill factor. Changed commitSubtree() recursion back to use a vector and waitForAll() because it seems to be lower overhead than ActorCollection.
2020-05-10 16:13:22 -07:00
f9518c3441
Merge pull request #3069 from alexmiller-apple/tls-connection-count
...
YOLO at reducing TLS connection count via doing monitorLeader on coordinators
2020-05-09 17:12:27 -07:00
69affebe40
merge master
2020-05-09 13:29:18 -07:00
2dfae85dc7
the delay for reads is about 15% of the total cost of the read, so start multiple reads with the same delay
2020-05-09 13:26:38 -07:00
02307ba7b6
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/DatabaseContext.h
2020-05-09 07:50:29 -07:00
f0f52fb2be
Merge branch 'master' into feature-small-endpoint
...
# Conflicts:
# fdbclient/StorageServerInterface.h
2020-05-08 16:37:35 -07:00
a833724322
Merge pull request #3078 from xumengpanda/mengxu/fr-circus-stall-PR
...
Performant restore: Various improvements based on circus test
2020-05-07 20:07:23 -07:00
a93c23d239
Resovle review comments
2020-05-07 15:06:59 -07:00
fbf436f45f
Various cleanup and knob adjustments.
2020-05-07 09:15:33 -07:00
8a6e177950
Merge remote-tracking branch 'upstream/master' into tls-connection-count
2020-05-05 16:49:36 -07:00
b1055a8501
Merge branch 'master' into transaction-tagging
2020-05-05 16:03:39 -07:00
f329164fb4
Merge pull request #2532 from dongxinEric/feature/hot-read-key-detection-part-2
...
Feature/hot read key detection part 2
2020-05-05 14:33:34 -07:00
c49b6756fe
FastRestoreApplier:Trace clear range op when it has too many for debug
2020-05-05 09:28:50 -07:00
2fec56e7e2
FastRestore:Logging for getReplyBatches
2020-05-04 20:12:59 -07:00
cd2176ac15
Adjust some knobs
2020-05-04 10:37:53 -07:00
36454bb3b8
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/MasterProxyInterface.h
# fdbclient/NativeAPI.actor.cpp
2020-05-04 10:23:25 -07:00
decf3e82b0
Fix various bugs and make sure to cleanup throttles from the database when they expire
2020-05-01 21:36:28 -07:00
43a63452d8
YOLO at reducing TLS connection count via doing monitorLeader on coordinators
2020-05-01 14:40:21 -07:00
8520c82ca0
updated dbinfo broadcast to minimize the number of possible WAN round trips
2020-05-01 13:20:26 -07:00
28178f356f
FastRestore:Minor knob change and revise comments
2020-05-01 10:47:44 -07:00
2c17fff6bb
FastRestore:Add batchIndex into metric event name
2020-05-01 10:05:33 -07:00
038f3834fc
Merge branch 'master' into mengxu/fr-code-improvement-PR
2020-05-01 09:26:29 -07:00
b3698ac921
Merge pull request #3060 from xumengpanda/mengxu/fr-agent-fix-PR
...
Fix fastrestore_agent errors when run in non-simulation mode
2020-04-30 22:46:39 -07:00
fb1c456a2d
FastRestore:Change default knob value
2020-04-30 19:15:32 -07:00
66228343f1
Merge branch 'master' into transaction-tagging
2020-04-30 08:12:03 -07:00
a0d67cac16
Merge branch 'master' into mengxu/fr-code-improvement-PR
2020-04-29 21:07:33 -07:00
6ada5359b8
Merge branch 'master' into transaction-tagging
2020-04-29 14:27:21 -07:00
7cebe743f9
A number of bug fixes of rare correctness errors
2020-04-29 13:50:13 -07:00
76fb345dd1
Merge branch 'master' into feature-tree-broadcast
...
# Conflicts:
# fdbrpc/FailureMonitor.actor.cpp
2020-04-29 09:51:22 -07:00
8dd05405eb
FastRestore:Guard knob with BUGGIFY
...
Prevent knob to be randomly set.
2020-04-28 16:05:40 -07:00
b80225dde0
Initial support for ramping load back up. Fix some logging. Update auto-throttles less frequently.
2020-04-28 15:50:45 -07:00
0ed70accfa
Reorganization of throttle storage in ratekeeper to support various auto-throttling related actions
2020-04-28 14:30:37 -07:00
b7f5f3be48
merge in master
2020-04-28 13:11:47 -07:00
41c517a5dd
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
2020-04-27 13:05:24 -07:00
ee5051792a
Add BUGGIFY back to DESIRED_TEAMS_PER_SERVER knob
...
Kudos to A.J. who found this bug.
2020-04-24 12:54:31 -07:00
239876351b
Add some initial auto-throttling. Move the definition of the priority enum to a more global place and use it for all transaction priorites (except in ClientLogEvents, because of serialization incompatibilites).
2020-04-24 11:31:16 -07:00
c87aa33941
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/go/src/fdb/generated.go
# documentation/sphinx/source/api-common.rst.inc
# documentation/sphinx/source/api-ruby.rst
# documentation/sphinx/source/release-notes.rst
# fdbclient/FailureMonitorClient.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/vexillographer/fdb.options
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/OldTLogServer_6_0.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# versions.target
2020-04-23 13:47:53 -07:00
0c84ad4bc6
Merge pull request #2917 from bnamasivayam/fail-slow-ss
...
Mark the storage servers that are continually lagging as unhealthy
2020-04-22 23:18:35 -07:00
9bf5c06d15
Adjust and knobify cost function for ops on the storage server
2020-04-22 14:39:32 -07:00
d0cc2a1ee4
added logging for parallel peeks on TLogs
2020-04-22 14:24:45 -07:00
434704fbd9
Various bug fixes
2020-04-22 12:28:51 -07:00
2960a2fe8a
FastRestore:Add knob to control parallelism in waiting requests
2020-04-19 21:34:11 -07:00
ba3e2af473
Merge commit '5288033bcfe40c3ade97c8bf2d04cf31b3f16cb1' into feature-tree-broadcast
2020-04-17 15:17:37 -07:00
dfec896438
Enforce a throttle limit. Don't count transaction tags on RK if the proxy has updated us in a while.
2020-04-17 11:48:02 -07:00
4a05910c6c
Buggify FASTRESTORE_GET_RANGE_VERSIONS_EXPENSIVE knob for true and false
2020-04-17 11:20:57 -07:00
2d9e9a0502
FastRestore:Use knob to guard the expensive way to get range versions
2020-04-17 10:02:58 -07:00
78d48a0dad
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbclient/Knobs.h
# fdbclient/NativeAPI.actor.cpp
# fdbclient/fdbclient.vcxproj
# fdbserver/MasterProxyServer.actor.cpp
2020-04-17 09:23:18 -07:00
7dd7406c59
Merge branch 'master' into feature/hot-read-key-detection-part-2
2020-04-16 14:54:05 -07:00
0fba8c47be
Checkpoint: Ratekeeper sets absolute limits for tag throttles and enforces them by distributing requests to proxies, who distribute them to clients.
...
A few refactorings.
2020-04-16 14:43:22 -07:00
dd951afe92
Merge pull request #2735 from ajbeamon/grv-proxy-perf-improvements
...
Improve proxy GRV latencies and responsiveness
2020-04-14 16:45:48 -07:00
08c8fac2af
Merge pull request #2943 from jzhou77/backup-fix
...
Buggify master recruit backup worker delay
2020-04-14 16:20:28 -07:00
9d6f2352d9
Merge commit 'cf01233f28a2c42908656a39f458a4475c1d44a3' into grv-proxy-perf-improvements
...
# Conflicts:
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/MasterProxyServer.actor.cpp
2020-04-14 13:49:09 -07:00
dbc9c23193
FastRestore:Loader:Send mutations at different versions in the same message to appliers
...
This increases the bandwidth sent from loaders to appliers.
2020-04-12 10:46:58 -07:00
87b8aae3e5
Buggify master recruit backup worker delay
2020-04-11 22:37:04 -07:00
8f78912483
knobified parameter
2020-04-11 20:54:17 -07:00
c0323f8a28
Merge commit '77f01193cbdf2ac5766131ffc8c71e8c88a0ed6b' into feature-tree-broadcast
2020-04-10 17:05:18 -07:00
07cc0a8d74
code cleanup
2020-04-10 17:02:11 -07:00
fe4bf3092f
Buggify DESIRED_TEAMS_PER_SERVER between 1 and 10
2020-04-10 09:43:01 -07:00
ebeca10bce
Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients.
2020-04-09 16:55:56 -07:00
36da61dd9c
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbclient/vexillographer/fdb.options
2020-04-07 21:12:14 -07:00
131ceeae1c
Adjust knobs.
2020-04-06 23:45:49 -07:00
73272fc72e
Version difference is now the diff between TLog versions and SS version.
2020-04-03 19:04:43 -07:00
2336f073ad
Checkpointing a bunch of work on throttles. Rudimentary implementation of auto-throttling. Support for manual throttling via fdbcli. Throttles are stored in the system keyspace.
2020-04-03 15:24:14 -07:00
884e92bb49
Atomically update dependent knobs
2020-04-01 15:18:49 -07:00
b1c3893d40
Fix some corner case bugs exposed by simulation.
...
In one case, when a SS joins the cluster and DD doesn't find any healthy server to form a team with the newly added server, then the SS does not get added to any team even when the other servers get healthy.
Another is an extreme case where a data center is down, and a SS in the active DC joins and then dies immediately but not before DD adds it to a destination team for a relocating shard which will result in DD waiting indefinitely for the dead data center to come back up for the cluster to be fully recovered.
2020-03-31 18:33:12 -07:00
40b17e1e9b
Remove a no longer unused knob
2020-03-26 13:04:00 -07:00
e0424a52f8
Merge branch 'master' into transaction-tagging
2020-03-25 08:23:11 -07:00
80d3fa1222
Add delay for master to recruit backup workers
...
This delay is to ensure old epoch's backup workers can save their progress in
the database. Otherwise, the new master could attempts to recruit backup
workers for the old epoch on version ranges that have already been popped. As
a result, the logs will lose data.
2020-03-20 20:15:08 -07:00
940bea102a
Add a knob to switch mutation logs for parallel restore
...
Knob FASTRESTORE_USE_PARTITIONED_LOGS, default is true to enable partitioned
mutation logs. Otherwise, old mutation logs are used.
2020-03-20 20:13:38 -07:00
26b7e02d4c
Some initial work to support tagging transactions and passing them around.
2020-03-20 11:23:11 -07:00
34415f82b3
Merge pull request #2832 from xumengpanda/mengxu/backup-code-review-PR
...
Buggify upload delay when backup worker upload data to blob
2020-03-19 21:42:28 -07:00
94276076de
BackupWorker:Buggify upload delay
...
Add questions to code as well.
2020-03-18 19:04:45 -07:00
58a9bfa78b
Merge pull request #2820 from dongxinEric/fix/1977/add-back-trace-event-flush-failure-report
...
Fix/1977/add back trace event flush failure report
2020-03-18 16:11:44 -07:00
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
ed4d02a3e4
Merge pull request #2812 from etschannen/feature-proxy-mem-limit
...
Limit the amount of requests the proxy can queue up in memory
2020-03-16 14:56:56 -07:00
d6d347f665
treat a tlog which takes a long time to create its disk queue as failed
2020-03-13 10:31:59 -07:00
243c268d9d
Limit the amount of requests the proxy can queue up in memory
2020-03-13 10:17:49 -07:00
5967ef5eab
Added back the changes that report trace log flush failures and fix the random crash
2020-03-12 14:34:19 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
e219c1671f
Merge branch 'release-6.2' into feature-dd-region-queue
...
# Conflicts:
# fdbserver/Knobs.h
2020-03-04 16:25:38 -08:00
6d6f184e2f
added a knob which reverts the new queue behavior
2020-03-04 16:23:49 -08:00
b7834b2995
Merge pull request #2774 from etschannen/feature-dd-repopulate-priority
...
Make the DD priority of populating a region lower than machine failures
2020-03-04 16:15:18 -08:00
39610d15f8
Revert this change since it somehow introduced a random crash detected on circus
2020-03-04 16:14:38 -08:00
6296465e07
Make the DD priority associated with populating a remote region lower than machine failures
2020-03-04 14:07:32 -08:00
2c6f82e1ab
FastRestore:Add unit name to threshold knob name
2020-03-02 10:52:44 -08:00
1ef4cb432b
Merge branch 'master' into mengxu/fast-restore-robust-and-visibility-PR-v2
2020-03-01 20:08:07 -08:00
ad9b3fb4a8
DD:Add trace for detailed relocate shard info
2020-02-29 13:45:10 -08:00
5c562e3714
FastRestore:Optimize memory usage in updateHeartbeatTime
...
Only allocate memory once.
Reset vector element value for each loop.
Also ensure FASTRESTORE_HEARTBEAT_DELAY not 0
2020-02-28 20:52:25 -08:00
01c1a15caf
FastRestore:Applier:Limit fetch keys number in a txn in getAndComputeStagingKeys
2020-02-28 16:53:36 -08:00
62b9043ff6
FastRestore:DB can be destroyed before master unlock it in simulation
...
Because retore roles run as workload in simulation,
they do not know when DB is destroyed by the backup and restore test workload.
So if DB is destroyed earlier than restore master unlocks DB, which is rare,
restore master should abort the unlocking DB step.
2020-02-28 14:25:58 -08:00
993c6e478e
Merge branch 'master' into grv-proxy-perf-improvements
2020-02-28 14:25:42 -08:00
13e72f7b3b
Merge pull request #2605 from dongxinEric/fix/1977/report-inability-to-flush-trace-log
...
Report inability to flush trace logs.
2020-02-27 12:36:55 -08:00
97d7eb49b5
FastRestore:Master:Report unavailable role periodically
...
Ping all restore roles and report unavailable ones.
2020-02-26 16:14:55 -08:00
ca726fc68e
FastRestore:Introduce OOM protection
...
An actor is schedulable to run if the current worker has enough resourc, i.e.,
the worker's memory usage is below the threshold;
Exception: If the actor is working on the current version batch, we have to schedule
the actor to run to avoid dead-lock.
Future: When we release the actors that are blocked by memory usage, we should release them
in increasing order of their version batch.
2020-02-26 14:09:18 -08:00
924d335aa7
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# flow/Knobs.cpp
# flow/Knobs.h
2020-02-25 18:25:19 -08:00
Alex Miller