02307ba7b6
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/DatabaseContext.h
2020-05-09 07:50:29 -07:00
ee0be88b99
consistency check can use a UIDtoTag map at a different version from the keyservers which could result in missing entries
2020-05-08 17:45:06 -07:00
6ada5359b8
Merge branch 'master' into transaction-tagging
2020-04-29 14:27:21 -07:00
a442565e13
more work towards shrinking locality
2020-04-18 21:29:38 -07:00
ba3e2af473
Merge commit '5288033bcfe40c3ade97c8bf2d04cf31b3f16cb1' into feature-tree-broadcast
2020-04-17 15:17:37 -07:00
78d48a0dad
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbclient/Knobs.h
# fdbclient/NativeAPI.actor.cpp
# fdbclient/fdbclient.vcxproj
# fdbserver/MasterProxyServer.actor.cpp
2020-04-17 09:23:18 -07:00
1439de37b5
Convert GetRangeLimits() -> TOO_MANY + ASSERT().
2020-04-12 18:23:14 -07:00
ce4493f679
many bug fixes
2020-04-10 13:45:16 -07:00
ebeca10bce
Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients.
2020-04-09 16:55:56 -07:00
6078fd1b18
Convert UID to Tag in keyServers to reduce txnStateStore size
2020-04-05 14:30:09 -07:00
e0424a52f8
Merge branch 'master' into transaction-tagging
2020-03-25 08:23:11 -07:00
26b7e02d4c
Some initial work to support tagging transactions and passing them around.
2020-03-20 11:23:11 -07:00
7f559bc712
Cleanup code and apply clang-format
...
Self code review
2020-03-16 15:08:32 -07:00
15c48b9e19
Add event for getDesired coordinators
2020-03-16 09:40:35 -07:00
0ef09539a9
addressMap[normalizedAddress]->address may not equal to normalizedAddress
2020-03-12 13:01:25 -07:00
1759d5c8c4
Apply clang-format
2020-03-12 10:18:53 -07:00
a9136f3f72
Add waitForUnreliableExtraStoreReboot to wait for extra store to reboot
2020-03-12 10:18:31 -07:00
d87ed92f78
checkForExtraDataStores:Fix compilation error
2020-03-11 09:59:11 -07:00
e0d2eca7a8
checkForExtraDataStores:Add coordinators into stateful process list
2020-03-10 23:38:30 -07:00
bd345f85db
ConsistencyCheck:Fix failue due to address inconsistency between process and worker
...
With TLS, a worker (or process) can have a TLS address and non-TLS address.
When a process is created in simulation, the primary address is TLS by default.
The non-TLS one is the TLS address port plus one.
In a connection between two workers, if their primary addresses do not enable
or disable TLS together, one worker will swap its primary address and secondary address
so that the TLS config of the two endpoints can match.
The swap can make the primary address no longer the TLS one that was created
when the process is created. And the swap only happens for worker instead of
process struct in simulation.
This swap can cause worker->address != process->address.
In checkForExtraDataStores actor, we use worker->address to check if a process
is killable and use the process->address to kill the process. The inconsistency
can cause simulation to kill a protected process that is not killable and leads
to simulation failure.
2020-03-10 21:07:16 -07:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
e422874758
fix: reboot does not work un unreliable processes
2020-02-21 14:29:42 -08:00
38a5511b96
additional simulation fixes
2020-02-11 15:52:06 -08:00
dcbce3593e
fixed TLS in simulation
2020-02-10 14:00:21 -08:00
acebfdc67b
Restore storage queue limit to 0 in consistency check
...
The storage queue is no longer going to be a problem failing tests. Now the
backup worker life cycle is tied with backup. So consistency check only happens
after the backup workload is done. Thus, we no longer need to save backup
progress when consistency check is running.
2020-01-22 19:43:40 -08:00
840e74d696
Allow storage server queue in consistency check
...
The backup worker needs to update its progress even during consistency check by
commit transactions to the database. Thus we can't really achieve zero storage
server queue. So add a limit of 10,000 to pass the consistency check.
2020-01-22 19:38:45 -08:00
8d3ef89540
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbclient/MutationList.h
# fdbserver/MasterProxyServer.actor.cpp
# versions.target
2019-11-14 15:49:56 -08:00
c26bb52979
Enable Consistency Checks for DD and RK.
2019-11-12 20:11:08 -08:00
f5282f2c7e
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc.
...
But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-12 14:22:36 -08:00
ae7ef12102
Fix nodiscard errors
2019-08-16 09:24:57 -07:00
5c98dcce6d
revert the proxy forwarding path, because it is no longer necessary as clients keep a persistent connection open with coordinators
2019-07-27 16:46:22 -07:00
94c66f8d58
Merge pull request #1738 from bnamasivayam/consistency-check-disable
...
Disable/Re-enable consistency check through a database key.
2019-07-18 10:56:02 -07:00
c348b3da51
After a proxy dies, it will remain alive for an additional 10 seconds to forward clients to the new proxies
2019-07-08 12:53:40 -07:00
7489f83a7f
Disable/Re-enable consistency check through a database key.
...
fdbcli has a new command 'consistencycheck' to disable/re-enable consistency check.
cluster_healthy metric in status becomes false if consistencycheck is disabled.
2019-06-20 21:38:45 -07:00
dccb9bc26d
fixed a number of correctness problems
2019-06-12 19:40:50 -07:00
d29c7e4c9b
Merge branch 'release-6.1' into merge-release-6.1-into-master
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/QuietDatabase.actor.cpp
# versions.target
2019-05-23 09:28:45 -07:00
9604452e50
mistakenly changed a quiet database parameter
2019-05-21 15:17:46 -07:00
4059d68348
fix: the tlog would not pop data from the disk queue after a storage server was removed, because the tag still exists in memory on the logs
...
fix: we could incorrectly make data durable if eraseMessagesFromMemory was in progress while running updatePersistentData
the quiet database check now ensure that tlogs have no more than 30 seconds of versions unpopped from the disk queue
2019-05-20 23:58:45 -07:00
b8e7fc1b84
Refactor: add std:: qualifier and use emplace_back
2019-05-17 09:38:50 -10:00
e0f76edf77
Merge pull request #1471 from AlvinMooreSr/release-6.1-merge
...
Merge Release 6.1 Into Master
2019-04-23 11:08:21 -07:00
ef04471a66
Fix more unused-variable warnings
2019-04-17 16:04:10 -07:00
43533b3d72
Don't validate the shard size estimate unless enough keys are sampled with a less than 100% probability.
2019-04-17 11:01:23 -07:00
04e9aa6afd
For small clusters that are growing quickly, it could happen that the rateLimit is set to a low value and it would take very long to read the entire database. Fix this by setting the rateLimit to the maximum allowed value if reading the entire database is taking a long time.
2019-04-10 17:13:37 -07:00
a7288e1325
Throw process_behind instead of future_version when all storage nodes on a team are behind. process_behind gets the same backoff behavior as not_committed. Add proxy_memory_limit_exceeded to the retryable predicate.
2019-04-08 14:21:24 -07:00
71e2fdafb8
Changes to ratekeeper camel case
2019-03-27 08:24:25 -07:00
10988f89d9
Code refactoring for ConsistencyCheck.actor.cpp
2019-03-23 11:06:43 -07:00
36ab852bb1
Merge branch 'master' into ratekeeper
...
# Conflicts:
# fdbserver/ClusterController.actor.cpp
2019-03-22 18:41:00 -07:00
f9aad46573
made use_provisional_proxies a transaction option
2019-03-19 18:44:37 -07:00
254c78053c
Fix a segfault error
...
After wait, ServerDBInfo may have changed. Using the old copy is wrong.
2019-03-15 22:11:13 -07:00
12ddd56698
Fix Ratekeeper and DataDistributor placement
...
Make sure both RateKeeper and DataDistributor are placed in the same data
center as the Master. Make sure only one RateKeeper is live in the cluster as
well.
2019-03-15 17:09:28 -07:00
A.J. Beamon