* Add multipart retry delay configuration. (#12076)
Add documentation to .h file.
Co-authored-by: stack <stack@duboce.com>
* Add a quickstart section on the bulkdump/bulkload feature to the user doc. (#12090)
* Fix assertion failure in fdbcli (#12095)
* Add Option to Turn Off Failure Injection to BulkLoad Process (#12096)
* add option to turn off failure injection to bulkload process in simulation
* nit
* DD BulkLoadTask Load Balance (#12103)
* bulkload load balance
* improve bulkload task submission efficiency
* address comments
* address comments
* cleanup
* Take a --proxy argument as backup_agent does. (#12110)
* Add "--proxy" to fdbserver (#12109)
* add --proxy to fdbserver
* fmt
* address comments
* use g_network to store proxy for s3client_ci
* Add 'ls' to s3client for the case where 'aws s3' is not installable (#12118)
* Add 'ls' to s3client for the case where 'aws s3' is not installable
-- i.e. production -- and we want a tool to test s3 connectivity.
Use it like this:
~/build_output/bin/s3client \
--tls-ca-file /etc/ssl/cert.pem \
--blob-credentials /path/to/credentials.json \
ls "blobstore://backup-us-west-2.s3.amazonaws.com/x?bucket=backup-us-west-2®ion=us-west-2"
* fdbclient/S3BlobStore.actor.cpp
* fdbclient/include/fdbclient/S3BlobStore.h
Make constructResourcePath method on endpoint so accessible making ls
URLs. Handling for encoded ampersands too. Allow 404 when requesting
resource that doesn't exist.
* fdbclient/S3Client.actor.cpp
* fdbclient/S3Client_cli.actor.cpp
Add in ls implementation.
* fdbclient/tests/s3client_test.sh
Add test for new ls facility.
* See if this fixes compile issue
---------
Co-authored-by: michael stack <stack@duboce.com>
* Ingest sst files rather than their keyvalue content (2nd attempt) (#12108)
* Ingest sst files rather than their keyvalue content.
* fdbclient/ServerKnobs.cpp
* fdbclient/include/fdbclient/ServerKnobs.h
Add BULK_LOAD_USE_SST_INGEST knob.
* fdbclient/include/fdbclient/IKeyValueStore.actor.h
Add ingestSSTFiles and supportSStIngestion.
* fdbserver/KeyValueStoreRocksDB.actor.cpp
* fdbserver/KeyValueStoreShardedRocksDB.actor.cpp
Implement ingestSSTFiles and supportSStIngestion
* fdbserver/storageserver.actor.cpp
If BULK_LOAD_USE_SST_INGEST and BulkLoadType::SST, ingest sst file
rather than read keyvalues.
* fdbclient/tests/fdb_cluster_fixture.sh
Use rocksdb instead of sqllite in tests.
* Revert "Ingest sst files rather than their keyvalue content."
This reverts commit 6d50bf15d3566de17fa6febf2afba0f220846ca6.
* ctests pass
* Restore ingest minus metric
* Add back ingestDuration metric
* Formatting
---------
Co-authored-by: michael stack <stack@duboce.com>
* Allow resource to be empty (#12129)
* RocksDB Do Compact After BulkLoad (#12130)
* rocksdb bulkload compact after load
* address comments
* address comments
* job overwrite task root path (#12138)
* BulkLoad Load Balance Based on DataMove Busy Map (#12137)
* bulkload load balance in ddqueue
* cleanup
* address comments
* nit
* bug fix
* nit
* address comments
* avoid scheduler busy loop
* address comments
* address comments
* avoid busy loop (#12157)
* First set of fixes... sizes and implement deleteResource (#12156)
Co-authored-by: michael stack <stack@duboce.com>
---------
Co-authored-by: Michael Stack <saintstack@users.noreply.github.com>
Co-authored-by: stack <stack@duboce.com>
* throttle hot shards
* expire throttled shards over time
* add backoff
* Parallelize messaging from RK to CP
* Obtain shards from a single SS
* handle expired transactions
* bump transaction_throttled_hot_shard
* Change SevError to SevWarn for CannotMonitorHotShardForSS
* Add log per request
Zhe Wang