* Add multipart retry delay configuration. (#12076)
Add documentation to .h file.
Co-authored-by: stack <stack@duboce.com>
* Add a quickstart section on the bulkdump/bulkload feature to the user doc. (#12090)
* Fix assertion failure in fdbcli (#12095)
* Add Option to Turn Off Failure Injection to BulkLoad Process (#12096)
* add option to turn off failure injection to bulkload process in simulation
* nit
* DD BulkLoadTask Load Balance (#12103)
* bulkload load balance
* improve bulkload task submission efficiency
* address comments
* address comments
* cleanup
* Take a --proxy argument as backup_agent does. (#12110)
* Add "--proxy" to fdbserver (#12109)
* add --proxy to fdbserver
* fmt
* address comments
* use g_network to store proxy for s3client_ci
* Add 'ls' to s3client for the case where 'aws s3' is not installable (#12118)
* Add 'ls' to s3client for the case where 'aws s3' is not installable
-- i.e. production -- and we want a tool to test s3 connectivity.
Use it like this:
~/build_output/bin/s3client \
--tls-ca-file /etc/ssl/cert.pem \
--blob-credentials /path/to/credentials.json \
ls "blobstore://backup-us-west-2.s3.amazonaws.com/x?bucket=backup-us-west-2®ion=us-west-2"
* fdbclient/S3BlobStore.actor.cpp
* fdbclient/include/fdbclient/S3BlobStore.h
Make constructResourcePath method on endpoint so accessible making ls
URLs. Handling for encoded ampersands too. Allow 404 when requesting
resource that doesn't exist.
* fdbclient/S3Client.actor.cpp
* fdbclient/S3Client_cli.actor.cpp
Add in ls implementation.
* fdbclient/tests/s3client_test.sh
Add test for new ls facility.
* See if this fixes compile issue
---------
Co-authored-by: michael stack <stack@duboce.com>
* Ingest sst files rather than their keyvalue content (2nd attempt) (#12108)
* Ingest sst files rather than their keyvalue content.
* fdbclient/ServerKnobs.cpp
* fdbclient/include/fdbclient/ServerKnobs.h
Add BULK_LOAD_USE_SST_INGEST knob.
* fdbclient/include/fdbclient/IKeyValueStore.actor.h
Add ingestSSTFiles and supportSStIngestion.
* fdbserver/KeyValueStoreRocksDB.actor.cpp
* fdbserver/KeyValueStoreShardedRocksDB.actor.cpp
Implement ingestSSTFiles and supportSStIngestion
* fdbserver/storageserver.actor.cpp
If BULK_LOAD_USE_SST_INGEST and BulkLoadType::SST, ingest sst file
rather than read keyvalues.
* fdbclient/tests/fdb_cluster_fixture.sh
Use rocksdb instead of sqllite in tests.
* Revert "Ingest sst files rather than their keyvalue content."
This reverts commit 6d50bf15d3566de17fa6febf2afba0f220846ca6.
* ctests pass
* Restore ingest minus metric
* Add back ingestDuration metric
* Formatting
---------
Co-authored-by: michael stack <stack@duboce.com>
* Allow resource to be empty (#12129)
* RocksDB Do Compact After BulkLoad (#12130)
* rocksdb bulkload compact after load
* address comments
* address comments
* job overwrite task root path (#12138)
* BulkLoad Load Balance Based on DataMove Busy Map (#12137)
* bulkload load balance in ddqueue
* cleanup
* address comments
* nit
* bug fix
* nit
* address comments
* avoid scheduler busy loop
* address comments
* address comments
* avoid busy loop (#12157)
* First set of fixes... sizes and implement deleteResource (#12156)
Co-authored-by: michael stack <stack@duboce.com>
---------
Co-authored-by: Michael Stack <saintstack@users.noreply.github.com>
Co-authored-by: stack <stack@duboce.com>
* Add multiparting to s3client.
Fix boost::urls::parse_uri 's dislike of credentialed blobstore urls.
* fdbclient/BulkLoading.cpp
Add blobstore regex to extract credentials before feeding the boost
parse_uri.
* fdbclient/include/fdbclient/S3BlobStore.h
* fdbclient/S3BlobStore.actor.cpp
Add cleanup of failed multipart -- abortMultiPartUpload l(s3 will do
this in the background eventually but lets clean up after ourselves).
Also add getObjectRangeMD5 so can do multipart checksumming.
* fdbclient/S3Client.actor.cpp
Change upload file and download file to do multipart always.
Retry too.
* fdbclient/S3Client_cli.actor.cpp
Add command line to trace rather than output.
* Address Zhe review
* More logging around part upload and download
* Undo assert that proved incorrect; restore the old length math
doing copy in readObject.
Cleanup around TraceEvents in HTTTP.actor.
* Undo commented out cleanup -- for debugging
* formatting
---------
Co-authored-by: stack <stack@duboce.com>
Fix object integrity check; original approach doesn't work when
serverside encryption is enabled (awz:kms).
* contrib/SimpleOpt/include/SimpleOpt/SimpleOpt.h
Address sanitizer was complaining about how SimpleOpt manipulates the
array of options. While memcpy inside a buffer is 'odd', it seems fine.
Its old code. Leaving it.
* fdbbackup/tests/s3_backup_test.sh
Pass in weed_dir rather than rely on fixture global (the latter didn't
work).
* fdbclient/ClientKnobs.cpp
* fdbclient/include/fdbclient/ClientKnobs.h
* fdbclient/include/fdbclient/S3BlobStore.h
Add a knob to ask for object integrity check on download from s3.
BLOBSTORE_ENABLE_OBJECT_INTEGRITY_CHECK replaces BLOBSTORE_ENABLE_ETAG_ON_GET
which doesn't work when serverside encodes content (found in testing).
* fdbclient/S3BlobStore.actor.cpp
Implement object integrity check on download. If
enable_object_integrity_check is set, we use sha256 in place of md5
as our hash. Removed a redundant 'verify' of md5 check.
* fdbclient/S3Client.actor.cpp
Remove unhelpful comments.
* fdbclient/S3Client_cli.actor.cpp
Add support for enable_object_integrity_check. This knob replaces
enable_etag_on_get which didn't work when awz:kms serverside
encryption was enabled.
Add error code on exit when exception.
* fdbclient/include/fdbclient/S3Client.actor.h
Move an include (address a review comment from previous commit).
* fdbclient/tests/aws_fixture.sh
Add an aws fixture of utility that can be shared.
* fdbclient/tests/bulkload_test.sh
Use imported log_test_result
* fdbclient/tests/s3client_test.sh
Add using s3 if available; otherwise, do seaweedfs.
* fdbclient/tests/seaweedfs_fixture.sh
WEED_DIR global doesn't work so have caller pass it in for each method
instead.
* documentation/sphinx/source/backups.rst
Minor edit. Add more examples making it clearer how to do S3
backup URLs in particular. Explain the 'trick' for omitting
key, secret, and token from URL instead picking them up from
the credentils file.
* fdbclient/S3Client_cli.actor.cpp
Minor cleanup of usage.
* fdbclient/ClientKnobs.cpp
* fdbclient/include/fdbclient/ClientKnobs.h
Add knob BLOBSTORE_ENABLE_ETAG_ON_GET
* fdbclient/S3BlobStore.actor.cpp
Optionally check etag (md5) volunteered by s3 against the
content we have downloaded and fail if not equal (TODO:
check the checksum after we've saved the content to the
filesystem -- would require good bit of a refactoring).
* fdbclient/S3Client.actor.cpp
Add deleteResource support.
* fdbclient/S3Client_cli.actor.cpp
Add COMMAND support; currently either 'cp' or 'rm'.
Set the knob blobstore_enable_etag_on_get to true by
default for s3client.
* fdbclient/tests/s3client_test.sh
Add clean up of resources written up to s3 at end of test.
(Awkward in bash)
fdbclient so can be used in fdbclient. Remove the copies of
BackupTLSConfig we had in place named BlobTLSConfig.*.
Keep the old name though it a little clunky.
.h file of "public" functions. Move the CLI processing and the TLSConfig
to standalone files; the former because of complaints of two main
functions when building simulation with combined S3Client and the latter
for clarity's sake -- one entity per file.
* fdbclient/S3Client.actor.h
"Interface" of S3Client public methods.
* fdbclient/S3Client.actor.cpp
Implementation of S3Client.actor.h
* fdbclient/BlobTLSConfig.cpp
* fdbclient/BlobTLSConfig.h
Move out of S3Cp/S3Client to its own file.
* fdbclient/S3Client_cli.actor.cpp
CLI for S3Client. Keep it separate because bundling the CLI with
the S3Client.actor becomes problematic building simulation tests
(linker complains of duplicated main, etc.)
* fdbclient/include/fdbclient/BlobTLSConfig.h
BlobTLSConfig (copied from BackupTLSConfig in fdbbackup).
* fdbclient/include/fdbclient/S3Client.actor.h
"Interface" for an S3 "Client" that runs on top of S3BlobStore.
Lists copy file and copy directory functions.
Zhe Wang