* Re-recruit log routers after failures
Log routers are stateless roles that can reconstruct its state after crash.
This is an attemp to avoid triggering recovery if one of log routers crashed.
To simplify the work, only the current generations of log routers are monitored
and re-recruited after crashes. Previous generations of log routers are not
handled in this change, as they are short lived and purged after recovery
reaches the fully_recovered state.
* Monitor log routers after full recovery
I.e., monitorAndRecruitLogRouters() waits for full recovery.
* Some cleanup
* Add WorkerCache for log routers
To avoid duplicated log routers running, though only one will be used (but it's
confusing when debugging).
20251114-230514-jzhou-59d6afe1e475c495
* Fix monitoring to happen after full recovery
20251115-041144-jzhou-4278fe608ed051b7
* Keep monitoring log routers before recovery completion
20251115-050228-jzhou-7c37cfb1d6e36ced
* monitorAndRecruitLogRouters detects recoveries
20251115-205132-jzhou-4d50f7c5914e883a
* Make monitorAndRecruitLogRouters long running
20251115-210426-jzhou-e63a8fcf26a76c81
* Recruit failed log routers in parallel
20251115-222239-jzhou-9eb2287e12c93f7a
* Rix replaced log router's begin version
Use the TLog's reply.popped version as its start version.
20251116-025329-jzhou-51a4def306038241
* clang-format fix
* Address review comments
20251120-221940-jzhou-28d51f8a0400377e compressed=True data_size=37463028 duration=8246235 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:51:39 sanity=False started=100000 stopped=20251120-231119 submitted=20251120-221940 timeout=5400 username=jzhou
* Add exponential backoff for log router re-recruitment
20251121-041700-jzhou-7126a109c1e39c76
* Fix crashes
20251121-043947-jzhou-6f373c1c64faa1b2
* Disable a verbose event
* Add CC_RERECRUIT_LOG_ROUTER_ENABLED to control this feature
20251215-220843-jzhou-70dad477b39640e9
* Test observability
* Tune recovery init timeouts in LowLatency test
* further tuning
* self review
* more tuning
* have an explicit disable knob for simulation
* minor formatting/spacing fix
* add clarifying comment
* add unit tests for getRecoverVersionUnicast()
* add version vector runtime check
* Add PartialAvailabilityPolicyFail, refactor tests to TagPartitionedLogSystemRecoveryTests.actor.cpp
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
starting from "max(KCV)" onwards. The current implementation allows
the version whose prevVersion is equal to "max(KCV)" to be skipped
and could cause the algorithm to pick an incorrect recovery version.
* More recovery observability for initializing_transaction_servers
* observability using new lib
* consistent naming
* address comments
* remove standalone, rely on upfront std::string copying
* More version vector/unicast related changes in order to get the
peek logic to work correctly during recovery:
- Ensure that non-buddy servers can return an empty version range
only if they are known to have been locked
- Make the MergePeekCursor follow the same logic that the
SetPeekCursor does when unicast is enabled
* - Do not try to reference known-locked-tLog-list (a version vector
specific data structure) when version vector is disabled.
* - Resolve a bug in the context of deciding when an empty version range
can be returned by a tLog
* - Code review related changes
* - More review related changes
* - Another review related change
* - More review related changes
* - More review related changes
* - Version vector/Unicast specific: Restart recovery if the list of
available tLogs change in such a way that the current in-progress
recovery could stall.
* - Code review related changes
* In the context of version vector/unicast, make log servers return
an empty version range (on peeks) only when it is correct to do so.
This is so the receiver will receive all versions (even though the
sender is sending an empty version range) that it is supposed to
receive.
Changes:
- Make the cluster controller collect the set of log servers participated
in recovery and propagate that information to the other processes.
- Extend the ServerPeekCursor to take a flag that tells the source log
server whether it can return an empty version range or not (in the context
of version vector/unicast), and make the source log server return an empty
version range only when this flag is set.
- Make the peek APIs set the flag appropriately when initializing
ServerPeekCursors.
- Also, take the internal logic used by SetPeekCursor and MergedPeekCursor
into account while initializing the above mentioned flag.
* - Update the member variable ("bestServer") after updating the argument
* - Review comments related changes
* - Review comments related changes
* Fix a race that can causes recovery to be stuck
When purging old generations, these no longer needed generations are removed in
the in-memory LogSystem data structure. Then the change is made durable on
coordinators.
If there is a recovery happened before the change is durable, and ServerDBInfo
broadcast is sent to the old tlogs and they can be displaced/removed. As a
result, the recovery will become stuck waiting for locking these old tlogs.
This PR updates the purging so that it never directly changes the in-memory data
structure and only modifies states on coordinators. So next recovery will pick
up the change.
* Add in-memory purging of old generation after writing coordinators
This allows old tlogs to be removed after CC purges them.
20250628-180019-jzhou-69511e1cbc01ab08
* Fix backup worker assertion failure on memory usage
pullAsyncData() can be cancelled thus not take() some of the memory used by
the in memory queue.
20250320-171737-jzhou-752fd8c45fdadd4a
100k tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.toml
20250320-172251-jzhou-3bc7db8e7ce5e1de
* Refactor such that messages in the queue have to reserve memory first
Thus, when we release the memory, it must have already been reserved.
100k ParallelRestoreNewBackupCorrectnessAtomicOp.toml
20250320-234748-jzhou-a159972dd0a72e03
20250320-235252-jzhou-54a6ae8c67873b59
* Fix a backup worker assertion failure
The pop version obtained from GRV proxy replies may go backwards, which can
cause assertion failure when pulling mutations, i.e., missing mutations. Fix
this bug and add an assertion that popVersion can go lower.
100k 20250321-012019-jzhou-01eb56938cf0a3fc
100k tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.toml
20250321-012144-jzhou-68181bb51aedbb5d
* Fix assertion failure of triggered version
The triggered version could be the same as popVersion.
20250321-025429-jzhou-a088c3f119331b93
100k tests/slow/ParallelRestoreNewBackupCorrectnessAtomicOp.toml
20250321-025532-jzhou-77e2dbd5fd035157
* Update fdbserver/BackupWorker.actor.cpp
Co-authored-by: Syed Paymaan Raza <1238752+spraza@users.noreply.github.com>
---------
Co-authored-by: Syed Paymaan Raza <1238752+spraza@users.noreply.github.com>
* - Extend the unicast based recovery algorithm to do the replication policy check
* - Review comments related changes
* - Review and compilation related changes
* During commits with version vector enabled, compute location list only once, as recalcuating could
generate a different random number, hence a different set of locations.
* Respond to review comments.
* Select replicas from locations returned from resolver.
* Respond to review comments
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* - When version vector is enabled use "min(DV)" as the recovery
version when trying to decide whether to restart recovery or not.
* - Address a review comment
* - Address a review comment
find the recovery version in the context of version vector. We will
need to extend the version vector recovery algorithm to do this
check in an efficient manner later.
the following ways:
- Make it use "max(KCV)", instead of "min(KCV)", as the default
recovery version
- Make it return the <max(KCV), recovery version> pair as output
- Make the invoker use the "max(KCV)" version returned by the
algorithm
- Address a bug related to the computation of recovery version (in
the case where a subset of log servers have empty
"unknownCommittedVersions" list)
And, make the invoker compute the max() and min() of recovery versions
of primary and satellite DCs based on the recovery versions computed
by the appropriate (with/without unicast) recovery version computation
algorithms.
* refactor management of cluster recovery version (crv)
- send the crv to tlogs as they rejoin cluster during recovery, not
just when an rv is computed.
- if a tlog has not yet received the crv, wait for it
* Propagate cluster recovery version to tlog for version vector recovery
Enable RECORD_RECOVER_AT_IN_CSTATE to store recovery version in cstate
* respond to review comments
* - dont sent rv to old tlog generations
- send crv after locking, if rv computed
* Remove CRV RPCs, have cursor pad beyond end tlog to RV
* Send end version in peek request to facility vv recovery
* Fix serialization in peek request interface.
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* Add assertions to code paths with txsTag
txsTag should be obsolete by now, since it's used in 6.1, which is no longer
supported for upgrade.
* Actually remove txsTag usage
20240926-225930-jzhou-7ed3304c415ae65e
* Remove more code
20240926-235242-jzhou-7ed3304c415ae65e
* Disable two verbose trace events
They can cause TraceTooManyLines errors.
* refactor to use struct rather than tuple for uncommitedVersions
* Document UnknownCommittedVersions struct
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* simulate more than one tlog
* Draft use cluster RV for tlogs in version vector
* add TestTLogRecovery2
* Respond to review comments
* Add assert
* Send clusterRV to all locked tlogs
* Fix typo on rebase
* add memory managed IdToInterf structure
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* - Recovery version computation when version vector unicast is enabled
* - Address a review comment
* - Modify code to not use "max(DV)" as the starting recovery version
* - Remove references to "max(DV)"
* - Address a review comment
* - Address PR review comments
* - Address a review comment
Jingyu Zhou