* Remove blob failure injections
Follow-up for the cleanup done at #12435. These functions are unused now.
* Fix an assertion failure in simulation
sim2 has "ASSERT(seconds >= -0.0001);" in delay() function, which was
triggering from the tlog code.
Reproduction:
-f ./tests/fast/SidebandSingle.toml -s 3567205446 -b on
* Optimize peekTracker lookups to use single try_emplace instead of find+operator[] pattern
* respond to review comments
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* Optimize peekVersionCounts/blockingPeekLatencies to use single map operation instead of three separate lookups
* Use underscore for unused binding variable
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* In the context of version vector/unicast, make log servers return
an empty version range (on peeks) only when it is correct to do so.
This is so the receiver will receive all versions (even though the
sender is sending an empty version range) that it is supposed to
receive.
Changes:
- Make the cluster controller collect the set of log servers participated
in recovery and propagate that information to the other processes.
- Extend the ServerPeekCursor to take a flag that tells the source log
server whether it can return an empty version range or not (in the context
of version vector/unicast), and make the source log server return an empty
version range only when this flag is set.
- Make the peek APIs set the flag appropriately when initializing
ServerPeekCursors.
- Also, take the internal logic used by SetPeekCursor and MergedPeekCursor
into account while initializing the above mentioned flag.
* - Update the member variable ("bestServer") after updating the argument
* - Review comments related changes
* - Review comments related changes
* ENABLE_VERSION_VECTOR_REPLY_RECOVERY can be T only if ENABLE_VERSION_VECTOR_TLOG_UNICAST is T
* Respond to review comments
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
is based over "TagData::popped" to decide how long to keep the disk
queue positions of versions in memory (instead of using the logic that
is based over "LogData::persistentDataVersion", which is applicable to
spill by value case).
* refactor management of cluster recovery version (crv)
- send the crv to tlogs as they rejoin cluster during recovery, not
just when an rv is computed.
- if a tlog has not yet received the crv, wait for it
* Propagate cluster recovery version to tlog for version vector recovery
Enable RECORD_RECOVER_AT_IN_CSTATE to store recovery version in cstate
* respond to review comments
* - dont sent rv to old tlog generations
- send crv after locking, if rv computed
* Remove CRV RPCs, have cursor pad beyond end tlog to RV
* Send end version in peek request to facility vv recovery
* Fix serialization in peek request interface.
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
allows "unknownCommittedVersions" to be kept in sync with the
containing LogData's "knownCommittedVersion", which is needed in
order to find the correct recovery version in the presence of
multiple recovery restarts.
* refactor to use struct rather than tuple for uncommitedVersions
* Document UnknownCommittedVersions struct
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* simulate more than one tlog
* Draft use cluster RV for tlogs in version vector
* add TestTLogRecovery2
* Respond to review comments
* Add assert
* Send clusterRV to all locked tlogs
* Fix typo on rebase
* add memory managed IdToInterf structure
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* - Recovery version computation when version vector unicast is enabled
* - Address a review comment
* - Modify code to not use "max(DV)" as the starting recovery version
* - Remove references to "max(DV)"
* - Address a review comment
* - Address PR review comments
* - Address a review comment
This PR includes a few stability fixes for Backup Worker
* Fixed memory bookkeeping issue in Backup Worker. Previously
it didn't release flow lock correctly when erasing messages.
* Added TLogServer fix to return 0 from poppedVersion() for
unrecognized log router tags.
We encountered a situation in simulation where the disk queue was in the following state
+------------+------------+
| page 1 | page 2 |
+------------+------------+
|rec |.......|rec |.......|
+------------+------------+
0..85 4096..4181
^. ^__ ^
popped. committed pushed
and we attempted to pop up to 4096, i.e. everything before page 2. This triggered
one of the assertions in the disk queue code which was meant to catch tlog logic
bugs where we pop too much.
The issue, though, is the accounting of the commit location in the disk queue.
While we only pushed records through position 85, we committed the entire page.
Attempts to pop everything before page 2 should have succeeded since we're not
attempting to pop any uncommitted data.
The solution is to fix the commit location accounting in the disk queue to round
up to the next page, to reflect the reality that we only commit entire pages.
This bug was discovered in the first place by introducing a delay into the commit
queue loop during simulation testing. That delay is included in this change.
We also noticed that getNextCommitLocation() was incorrect. Since there are no
users of that function, we've removed it entirely.
Jingyu Zhou