04d1217941
Track statistics about server-side request latency on each process, to include min, max, mean, and various percentiles.
2020-07-09 16:39:15 -07:00
69864c9f96
Make Spans not allocate heap memory
2020-07-09 11:49:33 -06:00
7afee53f4c
Clean up code that serves GRV through other proxies
2020-07-07 21:19:11 -07:00
0fbe7101c3
Revert "Revert "Request tracing""
...
This reverts commit 327cc31e35
2020-07-07 10:06:13 -06:00
832c6d0d65
Add comment about invariant master's committed version >= proxy's committedVersion
2020-07-06 12:34:15 -07:00
44331f50dc
Add back conditional report
2020-07-03 17:26:44 -07:00
286988c914
Report commit version later in the commit path
2020-07-02 16:14:50 -07:00
7546ed3edf
Remove trace events
2020-07-02 15:17:11 -07:00
323c65071d
Discard the condition to report commit version: only when commitVersion > committedVersion and leave trace events
2020-07-02 15:13:28 -07:00
31c70a873f
Report commit version to master as soon as possible
2020-07-02 15:13:28 -07:00
ed89d69916
Change order between reporting commit version and setting committedVersion
2020-07-02 15:13:28 -07:00
11b136c745
Various fixes to tag throttling:
...
* Master proxy reports transaction counts to ratekeeper for throttled tags only
* The ramp up behavior at the end of an auto-throttle was broken
* Fixed some issues with computing the initial transaction rate for auto-throttles
2020-06-30 16:24:41 -07:00
9ee21a1ae5
Fix getting read version while locked
2020-06-28 20:11:46 -07:00
b6beac4f83
Avoid reallocating for each batch in commitBatcher
2020-06-19 23:02:28 -07:00
85a284b13c
Use VectorRef::emplace_back in several places
2020-06-19 23:02:28 -07:00
8cd97a2be8
Resolve Evan's comments:
...
- Only report commit version when commit version is larger than the known committed version.
- Fix task priorities of [Get/Report]LiveCommittedVersion [request/reply].
- Fix some code style issues.
2020-06-17 10:02:25 -07:00
580ce6da74
Fix GetReadVersionReply.recentRequests
2020-06-16 22:45:57 -07:00
4dfb903a3a
tmp merge
2020-06-16 20:32:07 -07:00
327cc31e35
Revert "Request tracing"
2020-06-16 12:32:42 -07:00
bf524fc6f2
Added span trace in serveLiveCommittedVersion
2020-06-13 17:47:13 -07:00
f211a54593
Merged from upstream master
2020-06-13 16:47:12 -07:00
f8c457d74d
Minor fix against Meng's comments
2020-06-13 16:27:08 -07:00
2eb3aac651
Report commit version early to fix watcher tests on joshua
2020-06-12 00:10:33 -07:00
ff7354a7de
Recovered parallelism between the proxy-master rpc and its concurrent events and some minor fixes on formatting and variables naming
2020-06-11 14:07:37 -07:00
a47806a966
Fixed locked and metadataVersion in GetReadVersion
2020-06-10 15:55:23 -07:00
9ef93714ac
Address review comments
2020-06-10 15:48:49 -07:00
caabbec466
Spans for read and commit path
2020-06-10 09:59:41 -07:00
a038a02cdd
Serve GetReadVersion through master
2020-06-09 11:16:23 -07:00
c829b12bbf
defined spans for read requests
2020-06-08 17:26:15 -07:00
09c136e434
Framework for transaction tracing
2020-06-08 16:09:37 -07:00
1c35ad884f
Merge branch 'master' into mengxu/release-6.3-conflict-PR
...
Has conflict with master;
Next commit will fix the conflicts.
2020-05-25 12:01:49 -07:00
ced65cd30b
finished explicitly versioning everything stored in the database
2020-05-22 17:14:21 -07:00
d128252e90
Merge release-6.3 into master
2020-05-22 09:25:32 -07:00
f957dfabb5
clang-format
2020-05-22 05:25:13 -07:00
b44ec9c98a
Fix bug in reporting conflicting keys. Add read_conflict_range index mapping when we have multiple resolvers
2020-05-22 04:44:09 -07:00
7a09d016a6
Merge branch 'release-6.3' into merge-release6.3-into-master
2020-05-19 12:52:44 -07:00
6ce574f5ad
Merge remote-tracking branch 'upstream/release-6.3' into add-data-distribution-metrics
2020-05-17 23:36:52 -07:00
c2bc75516f
Merge branch 'release-6.3' of github.com:apple/foundationdb into features/trace-roles
2020-05-14 10:34:53 -07:00
ccaac162e2
Resolve performance concerns of nearly-no-op debugMutation being frequently called
...
This introduces unhygenic macro variants that inline a `ENABLED &&`
before the TraceEvent. This way, they get entirely compiled out unless
enabled.
Then rewrite all debugMutation uses via sed.
2020-05-13 18:44:15 -07:00
27da91ab9e
Merge remote-tracking branch 'upstream/master' into mutation-debugging
2020-05-13 12:51:44 -07:00
f17f00fdd5
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2020-05-10 22:33:38 -07:00
9d1df2aa58
fix: the first CPU estimate could be wrong enough to cause stalls on a newly recruited proxy
2020-05-10 17:50:09 -07:00
9d5e37f1a8
optimized decodeKeyServersValue on the proxy when loading all key ranges on startup
2020-05-09 16:54:35 -07:00
02307ba7b6
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/DatabaseContext.h
2020-05-09 07:50:29 -07:00
ef724bf939
Merge remote-tracking branch 'upstream/master' into add-data-distribution-metrics
2020-05-08 18:39:28 -07:00
5f9b127e56
Emit traces regularly about role assignment
...
We are currently emitting Role transition traces when a role starts and
when it ends. While this is useful for debugging, it doesn't work well
with tools that inject data and might potentially miss some trace lines.
We do decorate each trace lines with the roles assigned to that
particular process, however, this is not sufficient for tools that can
make use of the UID -> Role mapping
2020-05-08 16:27:57 -07:00
8166e1af8b
Storage server tags cannot change once used in the keyServer location map
2020-05-08 16:17:10 -07:00
36454bb3b8
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/MasterProxyInterface.h
# fdbclient/NativeAPI.actor.cpp
2020-05-04 10:23:25 -07:00
bb3d4b6b89
Add a bunch of TEST macros and some other little things
2020-05-04 10:11:36 -07:00
aed2d34bcb
Merge branch 'master' into feature-proxy-load-balance
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# flow/Knobs.cpp
2020-05-01 09:19:39 -07:00
6ada5359b8
Merge branch 'master' into transaction-tagging
2020-04-29 14:27:21 -07:00
627eb93dd9
Merge pull request #2941 from etschannen/feature-tree-broadcast
...
Implemented a tree broadcast for the proxy and cluster controller
2020-04-29 13:38:09 -07:00
110edc7453
Fix build on gcc9
2020-04-26 20:53:41 +00:00
239876351b
Add some initial auto-throttling. Move the definition of the priority enum to a more global place and use it for all transaction priorites (except in ClientLogEvents, because of serialization incompatibilites).
2020-04-24 11:31:16 -07:00
972be45d6f
Cleanup tags when possible on the proxy. Some minor cleanup.
2020-04-23 10:53:10 -07:00
434704fbd9
Various bug fixes
2020-04-22 12:28:51 -07:00
d5fb4d26fe
Send tag throttle updates from ratekeeper to proxy only when they change
2020-04-21 16:33:56 -07:00
b2c14611f3
Ratekeeper maintains its throttle data a bit differently, and now it aggregates the effects of multiple priorities before sending results to the proxy
2020-04-21 11:58:59 -07:00
ba3e2af473
Merge commit '5288033bcfe40c3ade97c8bf2d04cf31b3f16cb1' into feature-tree-broadcast
2020-04-17 15:17:37 -07:00
0e572c8833
Fix a couple correctness issues
2020-04-17 10:26:39 -07:00
1064ff483b
Fix merge issue
2020-04-17 09:37:34 -07:00
78d48a0dad
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbclient/Knobs.h
# fdbclient/NativeAPI.actor.cpp
# fdbclient/fdbclient.vcxproj
# fdbserver/MasterProxyServer.actor.cpp
2020-04-17 09:23:18 -07:00
6619a1a36a
Rename transaction tag map.
2020-04-17 09:06:45 -07:00
2b66dcd24a
Some more refactoring. Reduce what is sent from RK->MP->clients
2020-04-17 08:07:01 -07:00
0fba8c47be
Checkpoint: Ratekeeper sets absolute limits for tag throttles and enforces them by distributing requests to proxies, who distribute them to clients.
...
A few refactorings.
2020-04-16 14:43:22 -07:00
541c81a92a
Fix merge related issue.
2020-04-14 14:10:12 -07:00
9d6f2352d9
Merge commit 'cf01233f28a2c42908656a39f458a4475c1d44a3' into grv-proxy-perf-improvements
...
# Conflicts:
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/MasterProxyServer.actor.cpp
2020-04-14 13:49:09 -07:00
2e53c8c5e2
Merge pull request #2915 from tclinken/move-optimizations
...
Avoid unnecessary copies in PromiseStream
2020-04-13 21:15:16 -07:00
07cc0a8d74
code cleanup
2020-04-10 17:02:11 -07:00
ce4493f679
many bug fixes
2020-04-10 13:45:16 -07:00
29b2c2f3aa
Add hash to StringRef. Use unordered maps for storing tags. Create some helpful typedefs.
2020-04-10 12:54:59 -07:00
ebeca10bce
Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients.
2020-04-09 16:55:56 -07:00
36da61dd9c
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbclient/vexillographer/fdb.options
2020-04-07 21:12:14 -07:00
a1d8623e5f
Various changes to the throttling scheme, including most notably that clients now enforce the throttles with info they receive from the proxy.
2020-04-07 16:28:09 -07:00
a51c92854a
Merge branch 'master' into feature-tree-broadcast
...
# Conflicts:
# fdbserver/WorkerInterface.actor.h
# fdbserver/worker.actor.cpp
2020-04-06 21:09:44 -07:00
2a1bd97120
fix compilation errors
2020-04-06 20:58:43 -07:00
e682f6741c
Avoid unnecessary copies in PromiseStream
2020-04-06 17:15:03 -07:00
477d66b46d
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-05 23:09:36 -07:00
6078fd1b18
Convert UID to Tag in keyServers to reduce txnStateStore size
2020-04-05 14:30:09 -07:00
60f6edc3b5
Merge pull request #2860 from zjuLcg/report-conflicting-key-roll-forward
...
Report conflicting key roll forward
2020-03-30 17:33:56 -07:00
f7233bade7
Rename ParallelRestoreCorrectnessAtomicOpTinyData.txt by removing TinyData
2020-03-27 13:08:59 -07:00
113d0fb48b
Remove incorrect assertion
2020-03-27 12:13:30 -07:00
72e5891058
Clean up and rework the debugMutation API.
...
As a relatively unknown debugging tool for simulation tests, one could
have simulation print when a particular key is handled in various stages
of the commit process. This functionality was enabled by changing a 0
to a 1 in an #if, and changing a constant to the key in question.
As a proxy and storage server handle mutations, they call debugMutation
or debugKeyRange, which then checks against the mutation against the key
in question, and logs if they match. A mixture of printfs and
TraceEvents would then be emitted, and for this to actually be usable,
one also needs to comment out some particularly spammy debugKeyRange()
calls.
This PR reworks the API of debugMutation/debugKeyRange, pulls it out
into its own file, and trims what is logged by default into something
useful and understandable:
* debugMutation() now returns a TraceEvent, that one can add more details to before it is logged.
* Data distribution and storage server cleanup operations are no longer logged by default
2020-03-27 03:30:28 -07:00
64148469e8
clang-format the pr
2020-03-26 15:52:30 -07:00
289487559d
Revert "Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key""
...
This reverts commit 804fe1b22e
2020-03-24 18:11:15 -07:00
247ab84323
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-03-23 17:01:17 -07:00
26b7e02d4c
Some initial work to support tagging transactions and passing them around.
2020-03-20 11:23:11 -07:00
804fe1b22e
Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key"
...
This reverts commit 648dc4a933487d131b38
2020-03-19 21:34:28 -07:00
648dc4a933
Merge pull request #2257 from zjuLcg/report-conflicting-key
...
Report conflicting keys
2020-03-18 13:39:42 -07:00
e08f0201f1
merge release 6.2 into master
2020-03-17 12:51:47 -07:00
04b752b40a
Added additional logging related to memory errors (including in status)
2020-03-13 18:31:22 -07:00
a71e61f57b
fixed compiler issue
2020-03-13 18:22:38 -07:00
ebbf4490b3
use a Deque for each priority instead of a priority queue to improve CPU with large numbers of outstanding requests
2020-03-13 18:07:48 -07:00
243c268d9d
Limit the amount of requests the proxy can queue up in memory
2020-03-13 10:17:49 -07:00
02ee4f4c46
Update comments
2020-03-11 22:22:51 -07:00
7a76e9556d
Merge remote-tracking branch 'upstream/master' into report-conflicting-key
2020-03-04 11:24:39 -08:00
993c6e478e
Merge branch 'master' into grv-proxy-perf-improvements
2020-02-28 14:25:42 -08:00
1ec55f21f7
Add some comments for documentation; strengthen an assert.
2020-02-28 14:25:19 -08:00
924d335aa7
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# flow/Knobs.cpp
# flow/Knobs.h
2020-02-25 18:25:19 -08:00
6e7d2ff7dd
prevent the proxy from delaying too long based on an incorrect estimate of the compute time
2020-02-25 15:46:13 -08:00
A.J. Beamon