After a storageMetadataTracker detects that a storage server has been removed, the storageServerTracker should stop tracking this server. Without this fix, the
```
when(wait(storageMetadataTracker)) {}
```
will be triggered repeatedly, and the outer loop may busy looping printing trace lines, which eventually hit `TracedTooManyLines` error.
Test plan:
This failure was captured on snowflake/release-71.3 branch, but I believe the bug is also present on main. To verify that it fixes the issue, do the following:
1. Checkout branch snowflake/release-71.3
2. Profile: team ubsan
3. Commit: 507bbc98e2
4. Command: devRetryCorrectnessTest bin/fdbserver -r simulation -f tests/fast/SwizzledRollbackSideband.toml -s 1845441210 -b on --crash --trace_format json
5. Apply the simple fix, and rerun the test
In addition, we run a 100K Joshua ensemble.
20230714-220726-yajin-4f1585642a92bfd2, fail=0
* Terminate DD if SHARD_ENCODE_LOCATION_METADATA is not enabled and storage_engine_type is ShardedRocksDB.
* Fixed Error in non-main thread.
* Minor improvements.
Storage wiggle can cause consistency check to restart due to wrong_shard_server
error. This can sometimes repeatedly happen and cause consistency check timeout.
Reproduction:
seed: -f ./tests/fast/MutationLogReaderCorrectness.toml -s 575471711 -b on
commit: 2e62b577f at release-7.1
build: clang
In some rare cases, using the same failure time may prevent a certain size from
getting a team, thus cause data movement to become stuck.
Reproduction:
Seed: -f ./tests/rare/DcLag.toml -s 924031356 -b on
commit: 0975874b9 on release-7.3
build: gcc
prepare transactions
add DD Restore Preparing state; actor accept blob migrator requests
Refactor DDEnabledState and PrepareBlobRestoreReply
Improve the initialization, remove unused DDSharedContext method
move prepareBlobRestore to moveKeys
Make context->lock and DataDistributor.lock share the reference; change checkMoveKeysLock to checkPersistenMoveKeysLock; Add more debug trace
fix requestId assignment bug
Add DDEnabledState::sameId method
Throw movekeys_conflict after hybrid restore preparation to force reload
server list and shard mapping; format code; remove unused
methods,definition and comments
fix rebase conflicts
Make DD only load initial Data Distribution after enabled
use dd_config_changed error and throw it in serveBlobMigratorRequests
move empty range check before blob restore to the transaction lock database
Rename BlobRestore*
Yao Xiao