is based over "TagData::popped" to decide how long to keep the disk
queue positions of versions in memory (instead of using the logic that
is based over "LogData::persistentDataVersion", which is applicable to
spill by value case).
* refactor management of cluster recovery version (crv)
- send the crv to tlogs as they rejoin cluster during recovery, not
just when an rv is computed.
- if a tlog has not yet received the crv, wait for it
* Propagate cluster recovery version to tlog for version vector recovery
Enable RECORD_RECOVER_AT_IN_CSTATE to store recovery version in cstate
* respond to review comments
* - dont sent rv to old tlog generations
- send crv after locking, if rv computed
* Remove CRV RPCs, have cursor pad beyond end tlog to RV
* Send end version in peek request to facility vv recovery
* Fix serialization in peek request interface.
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
allows "unknownCommittedVersions" to be kept in sync with the
containing LogData's "knownCommittedVersion", which is needed in
order to find the correct recovery version in the presence of
multiple recovery restarts.
* refactor to use struct rather than tuple for uncommitedVersions
* Document UnknownCommittedVersions struct
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* simulate more than one tlog
* Draft use cluster RV for tlogs in version vector
* add TestTLogRecovery2
* Respond to review comments
* Add assert
* Send clusterRV to all locked tlogs
* Fix typo on rebase
* add memory managed IdToInterf structure
---------
Co-authored-by: Dan Lambright <hlambright@apple.com>
* - Recovery version computation when version vector unicast is enabled
* - Address a review comment
* - Modify code to not use "max(DV)" as the starting recovery version
* - Remove references to "max(DV)"
* - Address a review comment
* - Address PR review comments
* - Address a review comment
This PR includes a few stability fixes for Backup Worker
* Fixed memory bookkeeping issue in Backup Worker. Previously
it didn't release flow lock correctly when erasing messages.
* Added TLogServer fix to return 0 from poppedVersion() for
unrecognized log router tags.
We encountered a situation in simulation where the disk queue was in the following state
+------------+------------+
| page 1 | page 2 |
+------------+------------+
|rec |.......|rec |.......|
+------------+------------+
0..85 4096..4181
^. ^__ ^
popped. committed pushed
and we attempted to pop up to 4096, i.e. everything before page 2. This triggered
one of the assertions in the disk queue code which was meant to catch tlog logic
bugs where we pop too much.
The issue, though, is the accounting of the commit location in the disk queue.
While we only pushed records through position 85, we committed the entire page.
Attempts to pop everything before page 2 should have succeeded since we're not
attempting to pop any uncommitted data.
The solution is to fix the commit location accounting in the disk queue to round
up to the next page, to reflect the reality that we only commit entire pages.
This bug was discovered in the first place by introducing a delay into the commit
queue loop during simulation testing. That delay is included in this change.
We also noticed that getNextCommitLocation() was incorrect. Since there are no
users of that function, we've removed it entirely.
* Add server-side latency metrics for Resolver requests.
* Add separate resolver latency metrics for queue wait and compute time.
* Add histogram for queue depth observed on resolver (during metrics interval).
* Fix tlog latency measurement to use timer() instead of now().
These were lost, likely due to refactoring. Now TLogMetrics have meaningful
data like:
TLogMetrics ID=59ec9c67b4d07433 Elapsed=5 BytesInput=0 -1 17048 BytesDurable=47.4 225.405 17048 BlockingPeeks=0 -1 0 BlockingPeekTimeouts=0 -1 0 EmptyPeeks=1.6 2.79237 236 NonEmptyPeeks=0 -1 32 ...
We have a recent redesign that no longer required to pass tenant name to get encryption key, and also not allowing optional tenant mode for tenant-aware encryption. This PR clean up Redwood code to remove tenant map usage, and update various checks accordingly.
Changes:
* Cleanup TenantPrefixIndex in TenantAwareEncryptionKeyProvider and related logic in storage server and Redwood for passing the map around.
* Cleanup and update DecodeBoundaryVerifier the reflect the new design.
* A minor fix to writePages() that avoid a page that's default domain encrypted having a lower bound key belonging to a non-default domain.
* Fix TenantAwareEncryptionKeyProvider::getEncryptionDomain() returning wrong prefix long for system domain.
* A minor change to add a context string to IoTimeoutError.
Dan Lambright