6f7adb04a5
Send a version along with GetKeyServerLocationsRequests to allow the proxy to reject missing tenants more quickly. Change the commit proxy tenant map to be unversioned.
2022-03-08 13:33:56 -08:00
3ae98189af
Server processes send unknown_tenant responses back to clients, which is meant to be retried after updating the tenant cache. Fix bug where key selectors could be truncated after applying the tenant prefix.
2022-03-06 21:54:21 -08:00
72a34945ce
Add the ability to disable tenants. Server processes verify the ID of tenants being read or written.
2022-03-06 21:54:21 -08:00
c99c7cf55f
Apply tenant prefixes on the storage server. The commit proxy can serve tenant location requests. Fix some bugs.
2022-03-06 21:54:21 -08:00
5fa9d3e1b7
Add a tenant parameter to read and commit requests. Store a map of all tenants on commit proxy and storage servers. Add an option to require tenant mode.
2022-03-06 21:54:21 -08:00
bc7cc984b0
Fixing BGVerifyBalance test killing issues
2022-02-25 11:30:21 -06:00
250a88e682
Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement.
2022-02-24 12:25:52 -08:00
cddce79f25
Merge pull request #6359 from sbodagala/version-vector-prototype
...
Version vector prototype: Merge with the main branch
2022-02-08 15:56:52 -05:00
72e06369a4
Merge remote-tracking branch 'apple-upstream/main' into version-vector-prototype
2022-02-08 17:47:57 +00:00
54d96633c0
Do not sent metadata mutations to all resolvers
...
Let only resolver 0 to process metadata mutations and generate private mutations.
2022-02-07 11:49:56 -08:00
e6b02532a6
Merge pull request #6192 from sfc-gh-tclinkenbeard/improve-commitbatcher-performance
...
Make `VectorRefPreserializer` move constructor noexcept
2022-01-31 09:52:51 -08:00
9544379cdf
rebase
2022-01-20 11:12:33 -05:00
936bf5336a
Revert "Revert "Refactor: ClusterController driving cluster-recovery state machine" ( #6191 )
...
* Revert "Revert "Refactor: ClusterController driving cluster-recovery state machine""
Major changes includes:
1. Re-revert Sequencer refactor commits listed below (in listed order):
1.a. This reverts commit bb17e194d9d174bb2e0630b05b469c
2022-01-06 12:15:51 -08:00
54eb3b2e91
Disable static assertion in commitBatcher
2022-01-06 12:14:13 -08:00
648609d764
Make VectorRefPreserializer move constructor noexcept
2021-12-30 22:46:16 -08:00
30b05b469c
Revert "Refactor: ClusterController driving cluster-recovery state machine"
...
This reverts commit dfe9d184ff
2021-12-24 11:25:51 -08:00
dfe9d184ff
Refactor: ClusterController driving cluster-recovery state machine
...
At present, cluster recovery process consists of following steps:
1. ClusterController clusterWatchDatabase actor recruits
master/sequencer process.
2. Sequencer process implements the cluster recovery state machine,
responsible to recruit all other processes as well restore the
cluster state.
Patch proposes a scheme where the cluster recovery state machine
is implemented and driven by the ClusterController process instead
of the Sequencer process.
Advantages of the scheme could be:
1. Simplified design where ClusterController recruits "sequencer"
process like other worker processes compared to current scheme
where "sequencer" process gets special treatment. In newer scheme
sequencer is responsible for maintaining/providing
"committed version" (as expected).
2. ClusterController is responsible for worker processes recruitment,
the sequencer though orchestrating the recovery state machine, it
need to reachout to the ClusterController for recruiting worker
processes etc.
NOTE:
Patch has moved the recovery state machine code from
'sequencer' -> 'cluster-controller' process, however, necessary
updates were done for both functionality as well as performance
improvement reasons.
Next Steps:
Cluster recovery documentation will be updated in near future.
2021-12-22 14:06:27 -08:00
0ba77ea79b
Fix proxySnapCreate trace typo
2021-11-14 16:12:28 -08:00
5c258b089f
Fix written_tags not propagated to commit proxies
...
This is needed so that private mutation tags are propagated from Resolvers to
commit proxies. Otherwise, the master's VV won't be correct and can cause data
corruptions.
2021-11-11 11:26:13 -08:00
a8ba9bd021
Fix assertion of resolver returned mutation count
2021-11-09 11:45:53 -08:00
9f305d67a9
Fix backup mutation tags not counted in version vector
2021-10-22 08:43:20 -07:00
48a0ecd647
Merge pull request #5787 from dlambrig/integrate-PR5700
...
version vector / Calculate TPCV on resolvers
2021-10-20 16:35:44 -04:00
06f5401d14
Propagate written tags from proxy to sequencer when ENABLE_VERSION_VECTOR
...
is enabled, not when ENABLE_VERSION_VECTOR_TLOG_UNICAST is enabled.
2021-10-18 16:31:31 +00:00
eb814ce070
Remove dead code, check replies match for all resolvers
2021-10-18 10:23:08 -04:00
23062b892e
Calculate tpcv on resolvers
2021-10-15 16:40:00 -04:00
dd8840837d
Assert private mutation counts are the same from Resolvers
2021-10-15 12:13:20 -04:00
c6a9ac11a8
fix another conflict fdbserver/ApplyMetadataMutation.cpp
2021-10-15 09:48:33 -04:00
fc7325e923
Fix force recovery when generating private mutations
...
Resolver need to skip transactions after forceRecovery flag is true.
2021-10-15 09:47:23 -04:00
34e9932778
Separate \xff keyspace from keyResolvers
...
This is needed to make sure all Resolvers process all metadata mutations.
For commit proxy, we add all read/write conflict ranges for metadata mutations
to all resolvers.
We also send lock_aware flag for each transaction to resolvers.
2021-10-15 09:47:23 -04:00
fddf16534d
Send metadata mutations to all Resolvers
...
This is to make sure all Resolvers process them and make the same decision,
as well as maintain the same in-memory states, e.g., keyServers, serverKeys.
2021-10-15 09:47:23 -04:00
0dc9c607f4
Add knob PROXY_USE_RESOLVER_PRIVATE_MUTATIONS
...
To control proxy to use private mutations from resolvers or not.
2021-10-15 09:47:23 -04:00
c1d7b03087
CommitProxy uses Resolver's private mutations
...
Note the commit proxy still runs applyMetadataMutations for setting up memory
states, and no longer generates private mutations.
2021-10-15 09:47:23 -04:00
90a27bedf5
fix conflict fdbserver/ApplyMetadataMutation.cpp
2021-10-15 09:47:20 -04:00
9ecdbc39ee
Add private mutations to Resolver reply messages
2021-10-15 09:45:57 -04:00
b78dd3b9dd
Respond to comments 9/24
2021-09-30 11:51:57 -04:00
58e1888d8e
remove network hop by getting previous commit versions in GetCommitVersionRequest
2021-09-30 11:51:57 -04:00
2aa3b44d4e
Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
...
- Conflicts:
fdbserver/LogSystem.h
fdbserver/LogSystemConfig.h
fdbserver/TagPartitionedLogSystem.actor.cpp
- Files modified during merge:
modified: fdbserver/LogSystem.cpp
modified: fdbserver/LogSystemConfig.cpp
2021-09-17 19:36:18 +00:00
abf73047ca
Enforce std:: specifier rather than using namespace
2021-09-16 19:40:28 -07:00
8689e1f106
merge with master
2021-08-30 15:29:08 -04:00
e6b8a519c4
fixup! Make processTransactionStateRequestPart not blocking
2021-08-28 21:10:10 -07:00
74d8ca87d7
Refactor TxnStateRequest handler in commitProxyServerCore
2021-08-27 23:38:34 -07:00
69508b980f
modify comments to make clang-format and coverage tool play nice
2021-08-27 17:18:00 -07:00
2c788c233d
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-08-27 17:07:47 -07:00
a605fb3852
Merge pull request #5026 from sfc-gh-ljoswiak/fixes/alp6
...
Actor sampling
2021-08-11 13:44:17 -07:00
54c7036eaf
Move role UIDs for MutationTracking TraceEvents from various inconsistent detail fields into the TraceEvent UID field.
2021-08-10 01:52:36 -07:00
e5e8a56b66
Merge branch 'master' of https://github.com/apple/foundationdb into debugging-cleanup
...
# Conflicts:
# fdbbackup/FileDecoder.actor.cpp
2021-08-09 23:36:57 -07:00
faa4154a56
MutationTracking now uses a vector of keys to track. Removed "Mutation" detail from DEBUG_MUTATION() events because they are duplicates of the fields already logged in the returned MutationTracking event, which are now renamed and combined into "Mutation". Removed more toString() calls in TraceEvent detail values.
2021-08-09 23:30:45 -07:00
a081c0baa5
Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype
2021-08-05 22:40:32 +00:00
8609b45354
Add histograms to CommitProxyServer. ( #5299 )
2021-08-05 09:17:37 -07:00
7f3bc53858
- Address a bug
2021-08-02 19:35:59 +00:00
5dc9a97230
Merge branch 'master' into fixes/alp6
2021-08-01 20:42:52 -07:00
c74047c665
Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings
2021-07-28 11:51:02 -07:00
3eed4084e2
Merge branch 'master' into fixes/alp6
2021-07-27 11:26:53 -07:00
59d535149e
Merge branch 'master' into fixes/alp6
2021-07-27 10:07:18 -07:00
507c1f11e3
Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use.
2021-07-26 19:55:10 -07:00
48d4ea6d57
Merge pull request #5231 from sfc-gh-tclinkenbeard/availability-improvements
...
Throw `failed_to_progress` error when a role fails to make progress
2021-07-26 15:55:14 -07:00
d398377c4a
- Move the logic that populates written_tags (the tags that have been
...
affected by a transaction batch) to LogPushData
2021-07-26 16:27:19 +00:00
b9a22a61ef
Fix many -Wreorder-ctor warnings
2021-07-23 17:33:18 -07:00
1bb01ef591
use version vector in GetRange
2021-07-22 10:49:08 -04:00
6836e49073
Throw error when commitProxy gets stuck
2021-07-19 21:00:28 -07:00
3dc0c44187
Merge branch 'version-vector-prototype' into vv-79736536
2021-07-19 15:50:16 -04:00
8b2089e37b
- Fix a bug and address some review comments
2021-07-19 16:56:02 +00:00
65b743b5ef
respond to comments 8/15
2021-07-16 09:14:34 -04:00
81001edb2e
- Propagate version vector deltas between processes
2021-07-15 19:49:20 +00:00
b6f89df060
- Propagate the latest commit version of a storage server as part of read request.
...
Make storage server read at the specified version.
2021-07-15 19:48:42 +00:00
d07c8ce211
wait for prev per tlog
2021-07-15 08:44:48 -04:00
7c285aa4cf
wait for prev
2021-07-13 21:07:17 -04:00
d8122ebc09
Respond to review comments 7/8
2021-07-12 12:13:25 -04:00
17be3dc201
fix bug related to misunderstanding of tag replicas.
2021-07-12 11:46:52 -04:00
a107dd655e
tpcv messaging
2021-07-12 11:46:52 -04:00
1a20cf9579
Prevent commitProxyServer from modifying ServerDBInfo object
2021-07-11 23:29:36 -07:00
c700feaa6e
Address Dan's comments
2021-06-28 11:16:35 -07:00
66a97b76f1
Add metric to track the ratio of empty messages to tlogs
...
This metric is to understand the impact of batching on empty commit messages.
If the ratio is high, it means proxy sends many empty commit messages, which
can potentially be optimized for better performance. If the ratio is low, which
means the spread factor is large, thus we need to optimize the proxy to reduce
such a factor.
2021-06-24 21:26:09 -07:00
fcb8bd6475
Revert "Make the sim2 run loop match the behavior of the net2 run loop."
2021-06-22 14:50:01 -07:00
154332a94b
Merge branch 'master' of https://github.com/apple/foundationdb into feature-sim-time-batching
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2021-06-22 09:37:40 -07:00
65c49fc1e4
Merge pull request #4934 from sfc-gh-jslocum/tss_quarantine
...
Added TSS Quarantine and fixed TSS mapping bug in CommitProxy
2021-06-11 11:52:42 -07:00
c6d96c4124
Added TSS Quarantine and fixed TSS mapping bug in CommitProxy
2021-06-09 14:42:50 +00:00
08a5f17660
Merge branch 'master' of https://github.com/apple/foundationdb into feature-sim-time-batching
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
2021-06-08 10:04:06 -07:00
371a38e6e5
Merge remote-tracking branch 'origin/master' into remove-extra-copies
2021-06-07 10:26:06 -07:00
866f536983
Merge pull request #4888 from sfc-gh-tclinkenbeard/remove-fdbserver-includes
...
Remove fdbserver includes from fdbclient
2021-06-07 10:22:13 -07:00
486a04659f
Lazy inititialization
2021-06-04 15:01:18 -07:00
153de33f57
Revert "Merge pull request #4802 from sfc-gh-ljoswiak/revert/actor-lineage"
...
This reverts commit 6499fa178e1512631957
2021-06-04 13:31:55 -07:00
ac209b32fd
Addressing review comments
2021-06-03 15:31:16 +00:00
aca3941519
found bug in detecting first test in restart
2021-06-02 20:40:00 +00:00
b3e4f182ef
TSS Mapping Change
2021-06-02 17:30:09 +00:00
594e8944ae
Move RestoreWorkerInterface into fdbserver
2021-05-30 11:51:47 -07:00
f57f0d64f4
Merge branch 'master' into feature-sim-time-batching
...
# Conflicts:
# fdbserver/DataDistribution.actor.cpp
2021-05-20 09:09:35 -07:00
4ea760b2a9
Revert "Merge pull request #4136 from sfc-gh-mpilman/features/actor-lineage"
...
This reverts commit da41534618e6300905d6
2021-05-10 20:26:12 -07:00
f28ac955c3
Remove unnecessary temporary objects while growing objects of type std::vector<std::pair<A, B>>
2021-05-10 16:32:50 -07:00
5c2d7b6080
Create RangeResult type alias
2021-05-03 13:14:16 -07:00
d76b32da18
Annotate read paths on the server side
2021-04-20 15:10:01 -06:00
1d362d8869
Merge remote-tracking branch 'origin/master' into features/actor-lineage
2021-04-14 09:55:04 -06:00
feede1d2f6
Fix line length of test macro + comments to be within the 120 character limit
2021-04-13 10:48:52 -07:00
8e87627198
Merge remote-tracking branch 'origin/master' into features/actor-lineage
2021-03-19 17:19:02 -06:00
5b0c7bb789
fixed broken test macros
2021-03-19 13:22:45 -07:00
e1ebe2f487
clang-format
2021-03-19 13:17:39 -07:00
d7491a8f30
removed logging
2021-03-19 13:08:22 -07:00
0cb5c66fdf
Decerement commitBatchesMemBytesCount when rejecting commit batch
2021-03-19 18:30:50 +00:00
1a032f07dd
added logging
2021-03-18 15:49:08 -07:00
eb036b7b02
Merge remote-tracking branch 'origin/master' into features/actor-lineage
2021-03-17 11:59:54 -06:00
A.J. Beamon