Problem: FailedAssertion="value != endValue || endKey == maxWithPrefix.end" seen in a test cluster which regularly has a lot of data deleted (and hence, shard deletions).
Initial repro: PR #13177
The problems seems to be with how removeOldDestinations interacts with krmSetRangeCoalesting_, which has a very ominious but nonspecific comment about "use with care". We believe the correct solution is to have removeOldDestinations modify the entries one at a time, because other calls made on the same transaction read and update overlapping rows. This PR does that and also updates the comment in krmSetRangeCoalescing_ about what the actual requirements are.
This PR also includes an updated test (relative to PR#13177) to reproduce the problem directly in removeOldDestinations when run against the old version (it's left in the code for reference and as a cautionary example, enabled only behind a very loud ifdef).
The updated test case is complicated due to the need to run in simulation and to deal with various injected errors that are simply a fact of life there. All it really wants to do is arrange the race in removeOldDestinations() and ensure the outcome is correct (or the bug is reproduced, if it's present in removeOldDestinations).
removeOldDestinations is moved to the end of the file to avoid a "multiple definitions" compile error, which may possibly be a clang bug. There is a newly-added prototype in a header which is identical to the definition. Neither the agent or I could figure it out. Moving the definition of the function to after the call site in MoveKeys.cpp avoided the compile error.
Testing:
20260512-020318-gglass-30749afe25d28060 compressed=True data_size=37003190 duration=4183985 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:49:06 sanity=False started=100000 stopped=20260512-025224 submitted=20260512-020318 timeout=5400 username=gglass
* Update comment on krmSetRangeCoalescing(). Update removeOldDestinations() to call it in a way which does not generate corrupt metadata. Convert the test case from PR13177 to test removeOldDestinations() directly.
* Retry loops around transactions
* avoid getting hosed by short reads
* formatting
* Address copilot review comments
gxglass