Commit Graph

7 Commits

Author SHA1 Message Date
Trevor Clinkenbeard 3d852e8ead Merge remote-tracking branch 'refs/remotes/origin/main' into dev/tclinkenbeard/private-header-encapsulation-20260715 2026-07-20 13:01:51 -07:00
Akanksha Mahajan 96f56bb29a
Fix read-side overflow and simplify append API for large snapshot manifests (#13691)
* Fix read-side overflow and simplify append API for large snapshot manifests

  Follow-up to #13349, which fixed the write-side overflow when a snapshot manifest exceeds ~2 GB but left the read side and the API untidy. This PR addresses both.

  ### Changes

  **Simpler append API**
  There were two `append()` methods — one taking `int`, one `size_t` — and which ran depended on the argument type, which is easy to get wrong. Replaced with a single
  public `append()` that safely chunks any size, plus a clearly-named backend hook `appendImpl()` that each storage backend implements. No more overload ambiguity.

  **Read side fix**
  `readKeyspaceSnapshot` read the manifest into a buffer whose length is an `int`, so a manifest larger than 2 GB could truncate and crash on restore. It now reads into a
  `std::string` (which can exceed 2 GB) in chunks, matching the write side, and drops a redundant full copy of the manifest.

  **Knob rename**
  `BACKUP_MANIFEST_WRITE_CHUNK_SIZE` → `BACKUP_MANIFEST_CHUNK_SIZE`, since it now controls chunk size for both reads and writes.

  **Test**
  Added a unit test that reads a manifest back in many small chunks and verifies all range files and key ranges round-trip correctly.

  ### Notes
  - Range and log files are unaffected — they're already streamed in small blocks on both read and write.

* Addressed comments

* Fix clang tidy errors
2026-07-20 12:39:15 -07:00
Trevor Clinkenbeard cb9f79d484 Make AsyncFileBlobStore header private 2026-07-15 19:47:34 -07:00
Akanksha Mahajan 9ec2b21a80
Encryption backward compatibility changes and formatting (#13129)
* Encryption changes

* Addressed comments
2026-05-05 21:33:52 -07:00
Akanksha Mahajan 54795dcc94
Design changes to pass encryption_block_size to fdbbackup command and remove knob (#13023) 2026-04-30 13:32:14 -07:00
Trevor Clinkenbeard c127105f57 Improve header file encapsulation 2026-04-24 16:03:07 +00:00
Arnav Aggarwal ea921e6420
Introduce blob store interface and refactor S3 classes (#12936)
* Refactor blob store classes to use interface

* update to make some methods non-pure virtual

* Complete IBlobStoreEndpoint interface refactoring

Move doRequest, connect, returnConnection, updateSecret, writeEntireFile,
listObjects, and deleteRecursively from S3BlobStoreEndpoint to the shared
IBlobStoreEndpoint base class. Introduce provider hook methods
(setRequestHeaders, normalizeResourceForRequest, simulateRequestFailure,
processRequestFailure, preRetryCheck, extractCredentialFields) so that
the request loop is provider-agnostic.

Rename s3_backup_test.sh to backup_restore_test.sh and update CMake
to reflect the provider-agnostic naming.

Fix double proxy absolute-form rewrite bug where both
S3BlobStoreEndpoint::normalizeResourceForRequest and the base doRequest
would prepend http://host:port for proxy connections.

Fix getResourceURL duplication by having S3's override delegate to the
base class for shared knob-params/extra-headers logic.

* resolve review comments

* remove gcs auto region block

* clang format

* Enable +x bit.

* clang tidy
2026-04-17 09:17:15 -07:00