Tighten up options for bulk*. Compound 'local' and 'blobstore' as 'dump'/'load'. Ditto for 'history'.
Make it so 'bulkload mode' works like 'bulkdump mode': i.e. dumps current mode.
If mode is not on for bulk*, ERROR in same manner as for writemode.
Make it so we can return bulk* subcommand specific help rather than dump all help when an issue.
Make the commands match in the ctest
* fdbcli/BulkDumpCommand.actor.cpp
* fdbcli/BulkLoadCommand.actor.cpp
Print out the bulkdump description rather than usage so user
has a chance of figuring out what it is they entered incorrectly.
Make bulkdump and bulkload align by using 'cancel' instead of
'clear' in both and ordering the sub-commands the same for
bulkload and bulkdump. Add more help to the description.
Bulkload was missing mention of the jobid needed
specifying a bulkload.
* documentation/sphinx/source/bulkdump.rst
s/clearBulkDumpJob/cancelBulkDumpJob/
Co-authored-by: stack <stack@duboce.com>
It was disabled until we made it so the SS could
talk to s3, included in this PR.
Also finished the bulkload test. It only had the
bulkdump portion. bulkload support was recentlty
added so finish off the test here by adding bulkload
of the bulkdump and then verifying all data present.
Added passing knobs to the fdb cluster so available to the
fdbserver when it goes to talk to s3. Also added passing
SS count to start in fdb cluster.
* fdbclient/tests/fdb_cluster_fixture.sh
Add ability to pass multiple knobs to fdb cluster
and to specify more than just one SS.
* fdbserver/fdbserver.actor.cpp
Add --blob-server option and processing of FDB_BLOB_CREDENTIALS
if present (hijacked the unused, unadvertised --
blob-credentials-file).
* tests/loopback_cluster/run_custom_cluster.sh
Allow passing more than just one knob.
* fdbclient/BulkLoading.cpp
* fdbclient/include/fdbclient/BulkLoading.h
Added getPath
* fdbclient/S3BlobStore.actor.cpp
Fix bug where we were doubling up the first '/' on a path if
it had a root '/' already (s3 treats /a/b as distinct from
/a//b).
* fdbclient/S3Client.actor.cpp
Fix up of traceevent Types.
* fdbclient/tests/bulkload_test.sh
Enable being able to use s3 if available.
Pick up jobid when bulkdumping. Feed it to new bulkload
method. Add verification all data present post-bulkload.
* fdbserver/BulkLoadUtil.actor.cpp
Add support for blobstore.
* tests/loopback_cluster/run_custom_cluster.sh
Bug fix -- we were only able to pass in one knob. Allow
passing multiple.
* add bulkload job framework and fix bugs
* add BulkLoadChecksum, fix CI issue
* nits
* nits
* address comments
* mitigate perpetual wiggle to make sure DD can select a valid team to inject data
* fix submitBulkDumpJob and submitBulkLoadJob
* change remoteRoot to jobRoot
* add comments
* Improve BulkLoad/Dump implementation
* make bulkload test data folder inside simfdb folder
* simplify code
* use manifest in bulkdump metadata
* use manifest in bulkload
* apply bulkload fileset to bulkload and fix bugs of bytesampling value generation
* remove BulkDumpFileFullPathSet
* address comments
* address comments
* address comments
Zhe Wang