* Add metadata encoding audit, rollback support, and tests
Adds tooling and code to support safe SHARD_ENCODE_LOCATION_METADATA
rollback and migration verification:
audit_storage metadata_encoding:
New AuditType that scans keyServers and serverKeys to report encoding
format counts (old tag-based vs new UID-based). Reports migration
status: FORWARD COMPLETE / ROLLBACK IN PROGRESS / ROLLBACK COMPLETE.
DD startup rewrite (DDTxnProcessor.cpp):
When SHARD_ENCODE=false, clears stale DataMoveMetaData and rewrites
shard-encoded keyServers entries to old format. serverKeys entries are
left in place (readable in both formats, drain naturally).
MoveKeys graceful bail (MoveKeys.cpp):
Functions that require shard-encoding throw dd_config_changed instead
of asserting when they detect the knob flipped or DataMoveMetaData is
unexpectedly empty from a concurrent DD restart.
Error handling (DataDistribution.cpp, DDRelocationQueue.actor.cpp):
dd_config_changed added to normalDDQueueErrors to prevent SevError
logging for expected operational restarts.
Knob infrastructure (ServerKnobs.cpp):
SHARD_ENCODE randomization respects explicitlySetKnobs so TOML
overrides take effect in simulation tests.
Tests:
- CheckMetadataEncodingForward.toml: verifies new-format entries
- CheckMetadataEncodingOldPath.toml: verifies old-format entries
- ShardEncodeRollback.toml: full rollback test with Attrition,
Rollback (TLog recovery), data consistency check
Documentation:
- design/shard-encode-location-metadata.md: full feature description
* Address review feedback: fix doc, remove stale test refs, add pagination comment
- Fix doc: use correct command name (metadata_encoding not validate_metadata_encoding),
remove get_audit_status references (audit runs client-side, no persisted state)
- Remove stale ShardEncodeRollback-1/2.toml refs from CMakeLists (files don't exist)
- Remove metadata_encoding from GetAuditStatusCommand (not queryable after the fact)
- Add comment explaining pagination: rewrite loops via caller until all entries converted
* Add more on how metadata handling changes particular regards coalescing
* Formatting
* Address review feedback on shard-encode rollback
Bundled review fixes for the design doc and the DD rollback path:
- Drop the version field from the DataMoveMetaData ASCII diagram. The
field is serialized for evolvability but has no readers (only
invalidVersion is ever written; the metaData.version reads in the
tree are on CheckpointMetaData, a different struct). Removing it
from the diagram avoids implying semantics that arent there.
- Clarify the S3 path lifetime in the LOGICAL_BULKLOAD section: the
path lives in BulkLoadTaskState at \xff/bulkLoadTask/<range> and is
fixed for the lifetime of a given dataMoveId. Same dataMoveId on DD
restart is a continuation; a different dataMoveId for the same
range signals a new task and possibly a new path.
- Rename ROLLBACK IN PROGRESS to MIGRATION IN PROGRESS in
CheckMetadataEncodingCommand and the design doc. A snapshot of the
keyspace cannot distinguish forward-in-progress from rollback-in-
progress (both look like mixed format), so the previous label was
over-claiming a direction the audit cannot actually verify. The
status line now reports new-format counts only, with no "remaining"
framing.
- Define the four states reported by audit_storage metadata_encoding
(FORWARD COMPLETE, MIGRATION IN PROGRESS, ROLLBACK COMPLETE, NOT
STARTED) at the first mention, with the trigger condition for each.
Also note that ROLLBACK COMPLETE is what a cluster that never
enabled the knob will show, since the conditions dont distinguish
those cases.
- Expand the rewriteShardEncodedMetadata comment to describe the
bounded scope honestly: Phase 2 rewrites only the first 1000
keyServers entries per call, and clusters with more than that
remnant rely on shard movement / wiggle for the bulk rewrite (per
the "Migration for downgrade" section of the design doc). The
function is safe to call on every DD init when knob is false; the
no-op path is write-cost-free, three reads.
- Replace four ASSERTs in cleanUpSingleShardDataMove and the
raw{Start,CheckFetching,Finish}Movement dispatch helpers with
if-throws. cleanUpSingleShardDataMove now throws operation_cancelled
when krmGetRanges paginates (the data-move range was subdivided into
more shards than fit in one page; the caller's view is stale). The
three raw* helpers now throw dd_config_changed when the
MoveKeysParams field expected by the current SHARD_ENCODE_LOCATION_
METADATA branch is missing, matching the existing forward-direction
guards. All four guarded conditions become invalid when the knob
flips while a data-move's params are in flight: params built under
the old knob value reach dispatch under the new one. The asymmetry
was pre-existing but only surfaced now because ShardEncodeRollback
flips the knob mid-test.
Trevor Clinkenbeard