* Fix backup worker assertion failure on memory usage
pullAsyncData() can be cancelled thus not take() some of the memory used by
the in memory queue.
20250320-171737-jzhou-752fd8c45fdadd4a
100k tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.toml
20250320-172251-jzhou-3bc7db8e7ce5e1de
* Refactor such that messages in the queue have to reserve memory first
Thus, when we release the memory, it must have already been reserved.
100k ParallelRestoreNewBackupCorrectnessAtomicOp.toml
20250320-234748-jzhou-a159972dd0a72e03
20250320-235252-jzhou-54a6ae8c67873b59
* Fix a backup worker assertion failure
The pop version obtained from GRV proxy replies may go backwards, which can
cause assertion failure when pulling mutations, i.e., missing mutations. Fix
this bug and add an assertion that popVersion can go lower.
100k 20250321-012019-jzhou-01eb56938cf0a3fc
100k tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.toml
20250321-012144-jzhou-68181bb51aedbb5d
* Fix assertion failure of triggered version
The triggered version could be the same as popVersion.
20250321-025429-jzhou-a088c3f119331b93
100k tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.toml
20250321-025532-jzhou-77e2dbd5fd035157
* Update fdbserver/BackupWorker.actor.cpp
Co-authored-by: Syed Paymaan Raza <1238752+spraza@users.noreply.github.com>
---------
Co-authored-by: Syed Paymaan Raza <1238752+spraza@users.noreply.github.com>
* - Extend the unicast based recovery algorithm to do the replication policy check
* - Review comments related changes
* - Review and compilation related changes
* During commits with version vector enabled, compute location list only once, as recalcuating could
generate a different random number, hence a different set of locations.
* Respond to review comments.
* Select replicas from locations returned from resolver.
* Respond to review comments
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* - When version vector is enabled use "min(DV)" as the recovery
version when trying to decide whether to restart recovery or not.
* - Address a review comment
* - Address a review comment
find the recovery version in the context of version vector. We will
need to extend the version vector recovery algorithm to do this
check in an efficient manner later.
the following ways:
- Make it use "max(KCV)", instead of "min(KCV)", as the default
recovery version
- Make it return the <max(KCV), recovery version> pair as output
- Make the invoker use the "max(KCV)" version returned by the
algorithm
- Address a bug related to the computation of recovery version (in
the case where a subset of log servers have empty
"unknownCommittedVersions" list)
And, make the invoker compute the max() and min() of recovery versions
of primary and satellite DCs based on the recovery versions computed
by the appropriate (with/without unicast) recovery version computation
algorithms.
* refactor management of cluster recovery version (crv)
- send the crv to tlogs as they rejoin cluster during recovery, not
just when an rv is computed.
- if a tlog has not yet received the crv, wait for it
* Propagate cluster recovery version to tlog for version vector recovery
Enable RECORD_RECOVER_AT_IN_CSTATE to store recovery version in cstate
* respond to review comments
* - dont sent rv to old tlog generations
- send crv after locking, if rv computed
* Remove CRV RPCs, have cursor pad beyond end tlog to RV
* Send end version in peek request to facility vv recovery
* Fix serialization in peek request interface.
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* Add assertions to code paths with txsTag
txsTag should be obsolete by now, since it's used in 6.1, which is no longer
supported for upgrade.
* Actually remove txsTag usage
20240926-225930-jzhou-7ed3304c415ae65e
* Remove more code
20240926-235242-jzhou-7ed3304c415ae65e
* Disable two verbose trace events
They can cause TraceTooManyLines errors.
* refactor to use struct rather than tuple for uncommitedVersions
* Document UnknownCommittedVersions struct
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* simulate more than one tlog
* Draft use cluster RV for tlogs in version vector
* add TestTLogRecovery2
* Respond to review comments
* Add assert
* Send clusterRV to all locked tlogs
* Fix typo on rebase
* add memory managed IdToInterf structure
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* - Recovery version computation when version vector unicast is enabled
* - Address a review comment
* - Modify code to not use "max(DV)" as the starting recovery version
* - Remove references to "max(DV)"
* - Address a review comment
* - Address PR review comments
* - Address a review comment
The logic to determine the validity of a process joining a cluster now
belongs on the worker and the cluster controller. It is no longer
restricted to tlogs and storages, but instead applies to all processes
(even stateless ones).
Jingyu Zhou