* Add multipart retry delay configuration. (#12076)
Add documentation to .h file.
Co-authored-by: stack <stack@duboce.com>
* Add a quickstart section on the bulkdump/bulkload feature to the user doc. (#12090)
* Fix assertion failure in fdbcli (#12095)
* Add Option to Turn Off Failure Injection to BulkLoad Process (#12096)
* add option to turn off failure injection to bulkload process in simulation
* nit
* DD BulkLoadTask Load Balance (#12103)
* bulkload load balance
* improve bulkload task submission efficiency
* address comments
* address comments
* cleanup
* Take a --proxy argument as backup_agent does. (#12110)
* Add "--proxy" to fdbserver (#12109)
* add --proxy to fdbserver
* fmt
* address comments
* use g_network to store proxy for s3client_ci
* Add 'ls' to s3client for the case where 'aws s3' is not installable (#12118)
* Add 'ls' to s3client for the case where 'aws s3' is not installable
-- i.e. production -- and we want a tool to test s3 connectivity.
Use it like this:
~/build_output/bin/s3client \
--tls-ca-file /etc/ssl/cert.pem \
--blob-credentials /path/to/credentials.json \
ls "blobstore://backup-us-west-2.s3.amazonaws.com/x?bucket=backup-us-west-2®ion=us-west-2"
* fdbclient/S3BlobStore.actor.cpp
* fdbclient/include/fdbclient/S3BlobStore.h
Make constructResourcePath method on endpoint so accessible making ls
URLs. Handling for encoded ampersands too. Allow 404 when requesting
resource that doesn't exist.
* fdbclient/S3Client.actor.cpp
* fdbclient/S3Client_cli.actor.cpp
Add in ls implementation.
* fdbclient/tests/s3client_test.sh
Add test for new ls facility.
* See if this fixes compile issue
---------
Co-authored-by: michael stack <stack@duboce.com>
* Ingest sst files rather than their keyvalue content (2nd attempt) (#12108)
* Ingest sst files rather than their keyvalue content.
* fdbclient/ServerKnobs.cpp
* fdbclient/include/fdbclient/ServerKnobs.h
Add BULK_LOAD_USE_SST_INGEST knob.
* fdbclient/include/fdbclient/IKeyValueStore.actor.h
Add ingestSSTFiles and supportSStIngestion.
* fdbserver/KeyValueStoreRocksDB.actor.cpp
* fdbserver/KeyValueStoreShardedRocksDB.actor.cpp
Implement ingestSSTFiles and supportSStIngestion
* fdbserver/storageserver.actor.cpp
If BULK_LOAD_USE_SST_INGEST and BulkLoadType::SST, ingest sst file
rather than read keyvalues.
* fdbclient/tests/fdb_cluster_fixture.sh
Use rocksdb instead of sqllite in tests.
* Revert "Ingest sst files rather than their keyvalue content."
This reverts commit 6d50bf15d3566de17fa6febf2afba0f220846ca6.
* ctests pass
* Restore ingest minus metric
* Add back ingestDuration metric
* Formatting
---------
Co-authored-by: michael stack <stack@duboce.com>
* Allow resource to be empty (#12129)
* RocksDB Do Compact After BulkLoad (#12130)
* rocksdb bulkload compact after load
* address comments
* address comments
* job overwrite task root path (#12138)
* BulkLoad Load Balance Based on DataMove Busy Map (#12137)
* bulkload load balance in ddqueue
* cleanup
* address comments
* nit
* bug fix
* nit
* address comments
* avoid scheduler busy loop
* address comments
* address comments
* avoid busy loop (#12157)
* First set of fixes... sizes and implement deleteResource (#12156)
Co-authored-by: michael stack <stack@duboce.com>
---------
Co-authored-by: Michael Stack <saintstack@users.noreply.github.com>
Co-authored-by: stack <stack@duboce.com>
* Hash file before uploading. Add it as tag after successful
multipart upload. On download, after the file is on disk,
get its hash and compare to that of the tag we get from s3.
* fdbclient/CMakeLists.txt
Be explicit what s3client needs.
* fdbclient/S3BlobStore.actor.cpp
* fdbclient/include/fdbclient/S3BlobStore.h
Add putObjectTags and getObjectTags
* fdbclient/S3Client.actor.cpp
Add calculating checksum, adding it as
tags on upload, fetching on download,
and verifying match if present.
Clean up includes.
Less logging.
* fdbclient/tests/s3client_test.sh
Less logging.
* Make failed checksum check an error (and mark non-retryable)
---------
Co-authored-by: michael stack <stack@duboce.com>
* * fdbclient/S3Client.actor.cpp
Change field names so capitialized (convention)
Add duration as field to traces.
* fdbserver/BulkLoadUtil.actor.cpp
When the job-manifest is big, processing blocks
so much getBulkLoadJobFileManifestEntryFromJobManifestFile
fails.
* Make bulkload file reads and writes async and memory parsimonious.
In tests at scale, processing a large job-manifest.txt was blocking
and causing the bulk job to fail. This is part 1 of two patches.
The second is to address data copy added in the below when we
made methods ACTORs (ACTOR doesn't allow passing by reference).
* fdbserver/BulkDumpUtil.actor.cpp
Removed writeStringToFile and buldDumpFileCopy in favor of new methods
in BulkLoadUtil. Made hosting functions ACTORs so could wait on
async calls.
* fdbserver/BulkLoadUtil.actor.cpp
Added async read and write functions.
* fdbserver/DataDistribution.actor.cpp
Making uploadBulkDumpJobManifestFile async made it so big bulkloads
work.
* fix memory corruption in writeBulkFileBytes and fix read options in getBulkLoadJobFileManifestEntryFromJobManifestFile
* If read or write < 1MB, do it in a single read else do multiple read/writes
* * packaging/docker/fdb-aws-s3-credentials-fetcher/fdb-aws-s3-credentials-fetcher.go
Just be blunt and write out the credentials. Trying to figure when the
blob credentials have expired is error prone.
Co-authored-by: michael stack <stack@duboce.com>
Co-authored-by: Zhe Wang <zhe.wang@wustl.edu>
* Add multiparting to s3client.
Fix boost::urls::parse_uri 's dislike of credentialed blobstore urls.
* fdbclient/BulkLoading.cpp
Add blobstore regex to extract credentials before feeding the boost
parse_uri.
* fdbclient/include/fdbclient/S3BlobStore.h
* fdbclient/S3BlobStore.actor.cpp
Add cleanup of failed multipart -- abortMultiPartUpload l(s3 will do
this in the background eventually but lets clean up after ourselves).
Also add getObjectRangeMD5 so can do multipart checksumming.
* fdbclient/S3Client.actor.cpp
Change upload file and download file to do multipart always.
Retry too.
* fdbclient/S3Client_cli.actor.cpp
Add command line to trace rather than output.
* Address Zhe review
* More logging around part upload and download
* Undo assert that proved incorrect; restore the old length math
doing copy in readObject.
Cleanup around TraceEvents in HTTTP.actor.
* Undo commented out cleanup -- for debugging
* formatting
---------
Co-authored-by: stack <stack@duboce.com>
It was disabled until we made it so the SS could
talk to s3, included in this PR.
Also finished the bulkload test. It only had the
bulkdump portion. bulkload support was recentlty
added so finish off the test here by adding bulkload
of the bulkdump and then verifying all data present.
Added passing knobs to the fdb cluster so available to the
fdbserver when it goes to talk to s3. Also added passing
SS count to start in fdb cluster.
* fdbclient/tests/fdb_cluster_fixture.sh
Add ability to pass multiple knobs to fdb cluster
and to specify more than just one SS.
* fdbserver/fdbserver.actor.cpp
Add --blob-server option and processing of FDB_BLOB_CREDENTIALS
if present (hijacked the unused, unadvertised --
blob-credentials-file).
* tests/loopback_cluster/run_custom_cluster.sh
Allow passing more than just one knob.
* fdbclient/BulkLoading.cpp
* fdbclient/include/fdbclient/BulkLoading.h
Added getPath
* fdbclient/S3BlobStore.actor.cpp
Fix bug where we were doubling up the first '/' on a path if
it had a root '/' already (s3 treats /a/b as distinct from
/a//b).
* fdbclient/S3Client.actor.cpp
Fix up of traceevent Types.
* fdbclient/tests/bulkload_test.sh
Enable being able to use s3 if available.
Pick up jobid when bulkdumping. Feed it to new bulkload
method. Add verification all data present post-bulkload.
* fdbserver/BulkLoadUtil.actor.cpp
Add support for blobstore.
* tests/loopback_cluster/run_custom_cluster.sh
Bug fix -- we were only able to pass in one knob. Allow
passing multiple.
Fix object integrity check; original approach doesn't work when
serverside encryption is enabled (awz:kms).
* contrib/SimpleOpt/include/SimpleOpt/SimpleOpt.h
Address sanitizer was complaining about how SimpleOpt manipulates the
array of options. While memcpy inside a buffer is 'odd', it seems fine.
Its old code. Leaving it.
* fdbbackup/tests/s3_backup_test.sh
Pass in weed_dir rather than rely on fixture global (the latter didn't
work).
* fdbclient/ClientKnobs.cpp
* fdbclient/include/fdbclient/ClientKnobs.h
* fdbclient/include/fdbclient/S3BlobStore.h
Add a knob to ask for object integrity check on download from s3.
BLOBSTORE_ENABLE_OBJECT_INTEGRITY_CHECK replaces BLOBSTORE_ENABLE_ETAG_ON_GET
which doesn't work when serverside encodes content (found in testing).
* fdbclient/S3BlobStore.actor.cpp
Implement object integrity check on download. If
enable_object_integrity_check is set, we use sha256 in place of md5
as our hash. Removed a redundant 'verify' of md5 check.
* fdbclient/S3Client.actor.cpp
Remove unhelpful comments.
* fdbclient/S3Client_cli.actor.cpp
Add support for enable_object_integrity_check. This knob replaces
enable_etag_on_get which didn't work when awz:kms serverside
encryption was enabled.
Add error code on exit when exception.
* fdbclient/include/fdbclient/S3Client.actor.h
Move an include (address a review comment from previous commit).
* fdbclient/tests/aws_fixture.sh
Add an aws fixture of utility that can be shared.
* fdbclient/tests/bulkload_test.sh
Use imported log_test_result
* fdbclient/tests/s3client_test.sh
Add using s3 if available; otherwise, do seaweedfs.
* fdbclient/tests/seaweedfs_fixture.sh
WEED_DIR global doesn't work so have caller pass it in for each method
instead.
* fdbclient/ClientKnobs.cpp
* fdbclient/include/fdbclient/ClientKnobs.h
Add knob BLOBSTORE_ENABLE_ETAG_ON_GET
* fdbclient/S3BlobStore.actor.cpp
Optionally check etag (md5) volunteered by s3 against the
content we have downloaded and fail if not equal (TODO:
check the checksum after we've saved the content to the
filesystem -- would require good bit of a refactoring).
* fdbclient/S3Client.actor.cpp
Add deleteResource support.
* fdbclient/S3Client_cli.actor.cpp
Add COMMAND support; currently either 'cp' or 'rm'.
Set the knob blobstore_enable_etag_on_get to true by
default for s3client.
* fdbclient/tests/s3client_test.sh
Add clean up of resources written up to s3 at end of test.
(Awkward in bash)
* Improve BulkLoad/Dump implementation
* make bulkload test data folder inside simfdb folder
* simplify code
* use manifest in bulkdump metadata
* use manifest in bulkload
* apply bulkload fileset to bulkload and fix bugs of bytesampling value generation
* remove BulkDumpFileFullPathSet
* address comments
* address comments
* address comments
* init
* Add bulkdump to blobstore:// (s3)
* cmake/CompileBoost.cmake
Add boost url. Needed parsing blobstore:// urls.
* documentation/sphinx/source/bulkdump.rst
Minor edit to allow addition of blobstore target.
* fdbcli/BulkDumpCommand.actor.cpp
* fdbclient/BulkDumping.cpp
s/blobstore/s3/ -- more generic and aligns with
how backup/restore refers to "s3" thingies.
* fdbclient/include/fdbclient/S3Client.actor.h
* fdbclient/S3Client.actor.cpp
Add batch upload handler.
* fdbclient/tests/seaweedfs_fixture.sh
Add run seaweed method. Also look for
weed and if installed use it else download.
* fdbserver/BulkDumpUtil.actor.cpp
appendToPath does the right thing when passed an URL
Add bulkDumpTransportBlobstore_impl.
Add upload to blobstore.
* tests/loopback_cluster/run_custom_cluster.sh
Complain if unrecognized arguments.
* Add ctest for bulkload with simple bulkdump test for now.
* Add new test to ctest list
* fix bugs
* nit
* nits
* nits
---------
Co-authored-by: stack <stack@duboce.com>
fdbclient so can be used in fdbclient. Remove the copies of
BackupTLSConfig we had in place named BlobTLSConfig.*.
Keep the old name though it a little clunky.
.h file of "public" functions. Move the CLI processing and the TLSConfig
to standalone files; the former because of complaints of two main
functions when building simulation with combined S3Client and the latter
for clarity's sake -- one entity per file.
* fdbclient/S3Client.actor.h
"Interface" of S3Client public methods.
* fdbclient/S3Client.actor.cpp
Implementation of S3Client.actor.h
* fdbclient/BlobTLSConfig.cpp
* fdbclient/BlobTLSConfig.h
Move out of S3Cp/S3Client to its own file.
* fdbclient/S3Client_cli.actor.cpp
CLI for S3Client. Keep it separate because bundling the CLI with
the S3Client.actor becomes problematic building simulation tests
(linker complains of duplicated main, etc.)
* fdbclient/include/fdbclient/BlobTLSConfig.h
BlobTLSConfig (copied from BackupTLSConfig in fdbbackup).
* fdbclient/include/fdbclient/S3Client.actor.h
"Interface" for an S3 "Client" that runs on top of S3BlobStore.
Lists copy file and copy directory functions.
Zhe Wang