* feat: Add deterministic test failure injection for partial success testing
- Add SetTestFailurePredicate() virtual method to StorageBackendInterface
- Implement test failure injection in StorageBackendAdaptor and OffsetAllocatorStorageBackend
- Refactor TestPartialSuccessBehavior helper to use predicate-based injection
- Remove repetitive key creation code (use loops)
- Add PartialSuccess tests for both Adaptor and OffsetAllocator backends
- Update OutOfSpace test to handle partial success semantics (value==0)
This enables deterministic testing of partial success behavior without
relying on approximate space exhaustion, making tests more reliable.
* Addressed review comments
* [mooncake-store]: prevent orphaned bucket data files from leaking disk space.
Add cleanup mechanism for bucket data files that are left orphaned when
the process crashes between writing the data file and its metadata file.
Problem:
- BucketStorageBackend writes data file first, then .meta file
- If crash/failure occurs between these steps, data file is orphaned
- Init() only scans .meta files, never discovers orphans
- Orphaned files accumulate indefinitely, leaking disk space
Solution:
- Add immediate cleanup in WriteBucket() when metadata write fails
- Add orphan detection during Init() to clean up past crashes
- Make cleanup opt-in (disabled by default) via constructor flag
- Identify orphans by: numeric filename, no extension, no .meta pair
Changes:
- BucketStorageBackend: Add enable_orphan_cleanup parameter (default: false)
- WriteBucket(): Clean up data file if StoreBucketMetadata() fails
- Init(): Scan for and remove orphaned data files when enabled
- Add comprehensive test coverage for orphan cleanup
* [mooncake-store]: prevent orphaned bucket data files from leaking disk space.
Add cleanup mechanism for bucket data files that are left orphaned when
the process crashes between writing the data file and its metadata file.
Problem:
- BucketStorageBackend writes data file first, then .meta file
- If crash/failure occurs between these steps, data file is orphaned
- Init() only scans .meta files, never discovers orphans
- Orphaned files accumulate indefinitely, leaking disk space
Solution:
- Add immediate cleanup in WriteBucket() when metadata write fails
- Add orphan detection during Init() to clean up past crashes
- Make cleanup opt-in (disabled by default) via constructor flag
- Identify orphans by: numeric filename, no extension, no .meta pair
Changes:
- BucketStorageBackend: Add enable_orphan_cleanup parameter (default: false)
- WriteBucket(): Clean up data file if StoreBucketMetadata() fails
- Init(): Scan for and remove orphaned data files when enabled
- Add comprehensive test coverage for orphan cleanup
* [mooncake-store]: prevent orphaned bucket data files from leaking disk space.
Add cleanup mechanism for bucket data files that are left orphaned when
the process crashes between writing the data file and its metadata file.
Problem:
- BucketStorageBackend writes data file first, then .meta file
- If crash/failure occurs between these steps, data file is orphaned
- Init() only scans .meta files, never discovers orphans
- Orphaned files accumulate indefinitely, leaking disk space
Solution:
- Add immediate cleanup in WriteBucket() when metadata write fails
- Add orphan detection during Init() to clean up past crashes
- Make cleanup opt-in (disabled by default) via constructor flag
- Identify orphans by: numeric filename, no extension, no .meta pair
Changes:
- BucketStorageBackend: Add enable_orphan_cleanup parameter (default: false)
- WriteBucket(): Clean up data file if StoreBucketMetadata() fails
- Init(): Scan for and remove orphaned data files when enabled
- Add comprehensive test coverage for orphan cleanup
* [mooncake-store]: prevent orphaned bucket data files from leaking disk space.
Add cleanup mechanism for bucket data files that are left orphaned when
the process crashes between writing the data file and its metadata file.
Problem:
- BucketStorageBackend writes data file first, then .meta file
- If crash/failure occurs between these steps, data file is orphaned
- Init() only scans .meta files, never discovers orphans
- Orphaned files accumulate indefinitely, leaking disk space
Solution:
- Add immediate cleanup in WriteBucket() when metadata write fails
- Add orphan detection during Init() to clean up past crashes
- Make cleanup opt-in (disabled by default) via constructor flag
- Identify orphans by: numeric filename, no extension, no .meta pair
Changes:
- BucketStorageBackend: Add enable_orphan_cleanup parameter (default: false)
- WriteBucket(): Clean up data file if StoreBucketMetadata() fails
- Init(): Scan for and remove orphaned data files when enabled
- Add comprehensive test coverage for orphan cleanup
* [mooncake-store]: prevent orphaned bucket data files from leaking disk space.
Add cleanup mechanism for bucket data files that are left orphaned when
the process crashes between writing the data file and its metadata file.
Problem:
- BucketStorageBackend writes data file first, then .meta file
- If crash/failure occurs between these steps, data file is orphaned
- Init() only scans .meta files, never discovers orphans
- Orphaned files accumulate indefinitely, leaking disk space
Solution:
- Add immediate cleanup in WriteBucket() when metadata write fails
- Add orphan detection during Init() to clean up past crashes
- Make cleanup opt-in (disabled by default) via constructor flag
- Identify orphans by: numeric filename, no extension, no .meta pair
Changes:
- BucketStorageBackend: Add enable_orphan_cleanup parameter (default: false)
- WriteBucket(): Clean up data file if StoreBucketMetadata() fails
- Init(): Scan for and remove orphaned data files when enabled
- Add comprehensive test coverage for orphan cleanup
---------
Co-authored-by: Mahesh <maheshreddy.bapatu@nutanix.com>
Haode Du