It might have been easier to backport PR #12940 from main directly to release-7.4.
* Backport PR #12940 to release-7.3 (async status related) (#13068)
* Make status async and return before we timeout (#12940)
* Fix clusterGetStatus declaration for both actor compiler and clang-ide builds
The actor compiler adds const& to ACTOR params, so the source must use
value params. Clang-ide compiles .actor.cpp directly without the actor
compiler, so it needs a matching value-param declaration. Use
NO_INTELLISENSE guard to provide both.
* formatting
* Fix some misc backporting related issues
* Fix build for async status backport: remove tenant stuff, grey failure stuff, and update includes
* formatting
* remove dead code found in subsequent review
---------
Co-authored-by: Han Xu <hgcrpd@gmail.com>
* fix merge damage
* fix some more merge damage
* formatting
---------
Co-authored-by: Han Xu <hgcrpd@gmail.com>
* Backup encryption fix
* Add enable_read_cache option to be selected randomly to cover both code paths.
* Add randomization in ClientKnobs for testing
* Add multipart retry delay configuration. (#12076)
Add documentation to .h file.
Co-authored-by: stack <stack@duboce.com>
* Add a quickstart section on the bulkdump/bulkload feature to the user doc. (#12090)
* Fix assertion failure in fdbcli (#12095)
* Add Option to Turn Off Failure Injection to BulkLoad Process (#12096)
* add option to turn off failure injection to bulkload process in simulation
* nit
* DD BulkLoadTask Load Balance (#12103)
* bulkload load balance
* improve bulkload task submission efficiency
* address comments
* address comments
* cleanup
* Take a --proxy argument as backup_agent does. (#12110)
* Add "--proxy" to fdbserver (#12109)
* add --proxy to fdbserver
* fmt
* address comments
* use g_network to store proxy for s3client_ci
* Add 'ls' to s3client for the case where 'aws s3' is not installable (#12118)
* Add 'ls' to s3client for the case where 'aws s3' is not installable
-- i.e. production -- and we want a tool to test s3 connectivity.
Use it like this:
~/build_output/bin/s3client \
--tls-ca-file /etc/ssl/cert.pem \
--blob-credentials /path/to/credentials.json \
ls "blobstore://backup-us-west-2.s3.amazonaws.com/x?bucket=backup-us-west-2®ion=us-west-2"
* fdbclient/S3BlobStore.actor.cpp
* fdbclient/include/fdbclient/S3BlobStore.h
Make constructResourcePath method on endpoint so accessible making ls
URLs. Handling for encoded ampersands too. Allow 404 when requesting
resource that doesn't exist.
* fdbclient/S3Client.actor.cpp
* fdbclient/S3Client_cli.actor.cpp
Add in ls implementation.
* fdbclient/tests/s3client_test.sh
Add test for new ls facility.
* See if this fixes compile issue
---------
Co-authored-by: michael stack <stack@duboce.com>
* Ingest sst files rather than their keyvalue content (2nd attempt) (#12108)
* Ingest sst files rather than their keyvalue content.
* fdbclient/ServerKnobs.cpp
* fdbclient/include/fdbclient/ServerKnobs.h
Add BULK_LOAD_USE_SST_INGEST knob.
* fdbclient/include/fdbclient/IKeyValueStore.actor.h
Add ingestSSTFiles and supportSStIngestion.
* fdbserver/KeyValueStoreRocksDB.actor.cpp
* fdbserver/KeyValueStoreShardedRocksDB.actor.cpp
Implement ingestSSTFiles and supportSStIngestion
* fdbserver/storageserver.actor.cpp
If BULK_LOAD_USE_SST_INGEST and BulkLoadType::SST, ingest sst file
rather than read keyvalues.
* fdbclient/tests/fdb_cluster_fixture.sh
Use rocksdb instead of sqllite in tests.
* Revert "Ingest sst files rather than their keyvalue content."
This reverts commit 6d50bf15d3566de17fa6febf2afba0f220846ca6.
* ctests pass
* Restore ingest minus metric
* Add back ingestDuration metric
* Formatting
---------
Co-authored-by: michael stack <stack@duboce.com>
* Allow resource to be empty (#12129)
* RocksDB Do Compact After BulkLoad (#12130)
* rocksdb bulkload compact after load
* address comments
* address comments
* job overwrite task root path (#12138)
* BulkLoad Load Balance Based on DataMove Busy Map (#12137)
* bulkload load balance in ddqueue
* cleanup
* address comments
* nit
* bug fix
* nit
* address comments
* avoid scheduler busy loop
* address comments
* address comments
* avoid busy loop (#12157)
* First set of fixes... sizes and implement deleteResource (#12156)
Co-authored-by: michael stack <stack@duboce.com>
---------
Co-authored-by: Michael Stack <saintstack@users.noreply.github.com>
Co-authored-by: stack <stack@duboce.com>
* New restore consolidated commit
This change adds RestoreDispatchPartitionedTaskFunc to restore
from partitioned-format backup.
* ArenaBlock::totalSize parameter pass by ref
* Fix format issues identified by CI
Fix object integrity check; original approach doesn't work when
serverside encryption is enabled (awz:kms).
* contrib/SimpleOpt/include/SimpleOpt/SimpleOpt.h
Address sanitizer was complaining about how SimpleOpt manipulates the
array of options. While memcpy inside a buffer is 'odd', it seems fine.
Its old code. Leaving it.
* fdbbackup/tests/s3_backup_test.sh
Pass in weed_dir rather than rely on fixture global (the latter didn't
work).
* fdbclient/ClientKnobs.cpp
* fdbclient/include/fdbclient/ClientKnobs.h
* fdbclient/include/fdbclient/S3BlobStore.h
Add a knob to ask for object integrity check on download from s3.
BLOBSTORE_ENABLE_OBJECT_INTEGRITY_CHECK replaces BLOBSTORE_ENABLE_ETAG_ON_GET
which doesn't work when serverside encodes content (found in testing).
* fdbclient/S3BlobStore.actor.cpp
Implement object integrity check on download. If
enable_object_integrity_check is set, we use sha256 in place of md5
as our hash. Removed a redundant 'verify' of md5 check.
* fdbclient/S3Client.actor.cpp
Remove unhelpful comments.
* fdbclient/S3Client_cli.actor.cpp
Add support for enable_object_integrity_check. This knob replaces
enable_etag_on_get which didn't work when awz:kms serverside
encryption was enabled.
Add error code on exit when exception.
* fdbclient/include/fdbclient/S3Client.actor.h
Move an include (address a review comment from previous commit).
* fdbclient/tests/aws_fixture.sh
Add an aws fixture of utility that can be shared.
* fdbclient/tests/bulkload_test.sh
Use imported log_test_result
* fdbclient/tests/s3client_test.sh
Add using s3 if available; otherwise, do seaweedfs.
* fdbclient/tests/seaweedfs_fixture.sh
WEED_DIR global doesn't work so have caller pass it in for each method
instead.
* fdbclient/ClientKnobs.cpp
* fdbclient/include/fdbclient/ClientKnobs.h
Add knob BLOBSTORE_ENABLE_ETAG_ON_GET
* fdbclient/S3BlobStore.actor.cpp
Optionally check etag (md5) volunteered by s3 against the
content we have downloaded and fail if not equal (TODO:
check the checksum after we've saved the content to the
filesystem -- would require good bit of a refactoring).
* fdbclient/S3Client.actor.cpp
Add deleteResource support.
* fdbclient/S3Client_cli.actor.cpp
Add COMMAND support; currently either 'cp' or 'rm'.
Set the knob blobstore_enable_etag_on_get to true by
default for s3client.
* fdbclient/tests/s3client_test.sh
Add clean up of resources written up to s3 at end of test.
(Awkward in bash)
* Fix backup dryrun bug
Currently there is a out-of-scope issue, this change also adds
a knob to control whether to allow dryrun of backup
* fix another bug that misses a wait statement
---------
Co-authored-by: Hao <fdbflowguru@gmail.com>
* range lock framework
* improve the framework
* persist to txnStateStore
* fix bugs
* code clean
* code clean
* bug fix
* address comments
* add complex test workload and fix bugs found by the workload
* add workload correctness check and fix bugs
* code clean up
* add random range lock injection
* fix bugs in RandomRangeLock.actor.cpp
* enable random range lock injection in general workloads
* add rangelockcycle test
* disable random range lock in backup workloads
* nits
* add range lock ownership concept
* enable lock ownership to rangeLock
* api deal with tenant
* fix CI
* add test for multiple rangeLock owners
* nits
* address comments and renaming
* address comments
* ConsistencyCheckerUrgent repeated run
* address comments
* avoid trace SevError for TesterRecruitmentTimeout unless it keeps failure for over 1 day
* address comments
* address comments
* - Compare storage replicas on reads (in "loadBalance()")
* - Do consistency check on reads in loadbalance
* - Do replica consistency check in the case where loadBalance issues
requests to multiple storage servers
* - Address a state variable related bug
* - Code formatting
* - API simplification
* - Simplify code
* - Code formatting
* - Address a review comment
* acs framework
* code refactor and fix bugs
* add ss crash loop protector
* use sharedptr instead of raw pointer
* fixed critical bugs and add provate mutation acs to the framework
* enable ACS for all mutations except for clear serverTag mutation and fix bugs
* fix restarting tests
* refactor code and fix bugs
* fix AccumulativeChecksumState toString
* fix bugs
* allow all mutations in acs and fixed bugs
* fix bugs and code cleanup
* code clean up for adding recovery support
* simplify code and support recovery
* clear acs state at ss
* fix bug
* terminate validator if ss will be removed in the current batch
* simplify code
* add trace
* address comments
* optimize code
* deep copy when adding mutation to acs validator
* warp encode and decode persist acs key
* make acstable private
* remove unless func
* remove unless func
* remove epoch in ACS validator
* add acs mutation counter in SS metrics
* code cleanup and make knob check better
* make mutation buffer global
* simplify code
* add comments
* make knob randomly set
* address comments
* ss reboot after acs mismatch found
* cherry-pick-distributed-consistency-checker
* code cleanup
* refactor code, decouple consistencyCheckerUrgent and consistency checker
* fix workload for consistencycheckurgent
* add new consistencycheckurgent role type
* fix CI
* address comments
We learnt that a new connection needs to be made for each HTTP
request through proxy to AWS S3, thus it would fail when try
to re-use the connections and see retryable errors.
Meanwhile, delay between each retry grow 2x each time it failed,
if the connection pool has a larger size, the delay can be long.
As a result, this change adds Knobs to cap the max delay of
retryable errors, including one for general retryable errors,
and the other only for connection failure error.
This change also adds more logging for debugging.
Description
Given Configurable encryption has been checked in and being tested via
simulation for more than a month and also to avoid penalty of accessing
KNOBS in inline commit path, patch retires the KNOB and make
ConfigurationEncryption default EaR mode for FDB.
BlobCipher still supports the old format header and encryption semantics,
will remove the dead code as a followup PR.
Testing
devRunCorrectness - 100K
* EaR: REST KMS fixes - encryption integration testing
Description
Major changes:
1. Multiple fixes observed while performing integration end-to-end
testing for Encryption at-rest feature.
2. Improve REST module logging. Introduced FLOW_KNOBS->REST_LOG_LEVEL
to have more granular control of feature logging disconnected from
the cluster log level.
Testing
Integration testbed:
1. Run fdbserver standalone
2. Run external KMS http-server to serve encryption key fetch requests
* EaR: RESTClient HTTP compliance, fix json request content type
Description
diff-1: Address review comments
RESTClient is responsible to handle FDB <-> KMS communication
for Encryption and other usecases. By design, it only supports
"secure connection" i.e. "https"; however, it seems there is a
need to expand the module to support "http" connection,
for instance: test and dev deployments for instance.
However, given RESTClient gets involved in handling high
sensitive contents such as: plaintext "encryption cipher
from a KMS", the feature is guarded using
CLIENT_KNOB->REST_KMS_ENABLE_NOT_SECURE_CONNECTION which is
settable using FDBServer command line argument
"--kms-rest-enable_not_secure_connection" (boolean)
Testing
Deployed a standalone fdbserver and communicate with a
simple "http" server
gxglass