Commit Graph

576 Commits

Author SHA1 Message Date
He Liu 00203c8732
Validate Storage part II (#8471)
* Implemented AuditUtils.actor.cpp

Moved AuditUtils to fdbserver/

* Persist AuditStorageState.

* Passed persisted AuditStorageState test.

* Added audit_storage_error to indicate a corruption is caught.

Throw/Send audit_storage_error when there is a data corruption.

Added doAuditStorage() for resuming Audit.

* Load and resume AuditStorage when DD restarts.

* Generate audit id monotonically.

* Fixed minor issue AuditId/Type was not set.

* Adding getLatestAuditStates.

* Improved persisted errors and added AuditStorageCommand.actor.cpp for
fdbcli.

* Added `audit_storage` fdbcli command.

* fmt.

* Fixed null shared_ptr issue.

* Improve audit data.

* Change DDAuditFailed to SevWarn.

* Sev.

* set SERVE_AUDIT_STORAGE_PARALLELISM to 1.

* Moved AuditUtils* to fdbclient/.

* Added getAuditStatus fdbcli command.

* Refactor audit storage fdb cli commands.

* Added auditStorage in sim.

* Cleanup.

* Resolved comments.

* Resolved comments.

* Test disabling audit for sims.

* Cleanup.

Co-authored-by: He Liu <heliu@apple.com>
2023-01-15 21:46:14 -08:00
Ankita Kejriwal fe06114bcd Update the help text for `quota` command 2022-12-14 11:01:33 -08:00
FoundationDB CI 86d6106dc1
format source code after switch to clang 15 2022-12-08 17:26:45 +00:00
Daniel Luan 4be05e5a5d Upgrade C++ Standard to 20 2022-12-06 14:19:06 -08:00
Hui Liu 5834517570 Add fdbcli blobrestore to start the full restore 2022-11-11 08:32:23 -08:00
Kevin Hoxha 2dac017ea2 fdbcli: Cancel checkStatus actor when prompting for description in setknob 2022-10-20 21:31:52 -07:00
sfc-gh-tclinkenbeard 9a1876dd11 Update comment for fdbcli quota command 2022-10-14 12:02:14 -07:00
sfc-gh-tclinkenbeard 3827b1ee62 Merge remote-tracking branch 'origin/main' into add-quota-clear-command 2022-10-14 11:55:45 -07:00
Kevin Hoxha ff1b2df8f6 fdbcli: Add options for knob management
- setknob <knob_name> <knob_value> [config_class]
- getknob <knob_name> [config_class]
- Added new option to begin to specify if it's a configuration txn. Syntax is begin [config-txn]
- Added utility function for converting tuples to string
- Added knobmanagment test in fdbcli_tests.py
2022-10-11 15:32:01 -07:00
sfc-gh-tclinkenbeard d67f3ca3b9 Update documentation for quota clear command 2022-10-09 19:43:44 -07:00
Trevor Clinkenbeard ef13985feb
Merge pull request #8320 from sfc-gh-tclinkenbeard/read-write-fungible
Make read and write quotas fungible
2022-09-28 10:59:37 -07:00
Junhyun Shim fd621c0c86 Make TLSConfig check against coordinator addresses one-way
TLS parameters may be loaded by default system paths, which would break
non-TLS cases
2022-09-27 20:57:38 +02:00
sfc-gh-tclinkenbeard 7fc5c196c4 Make read and write quotas fungible 2022-09-25 21:00:11 -07:00
Josh Slocum 28f3cacfd7
Merge pull request #8285 from sfc-gh-jslocum/bg_history_fdbcli
Adding blobkey history fdbcli debug command
2022-09-22 16:56:59 -05:00
Josh Slocum 36df430b14 Merge branch 'main' into bg_history_fdbcli 2022-09-22 15:49:07 -05:00
A.J. Beamon 97a325adab Add an fdbcli command to get tenant group metadata 2022-09-22 13:24:21 -07:00
Josh Slocum 21c627ba97 Adding blobkey history fdbcli debug command 2022-09-22 13:03:53 -05:00
A.J. Beamon 64c8e0e2b1 Convert fdbcli tenant commands to a single command with subcommands. Update the structure of fdbcli tenant tests. 2022-09-21 16:22:58 -07:00
A.J. Beamon 4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
sfc-gh-ngoyal 1bd97fe628
Recruit new singleton for consistency checker. (#5804)
* Recruit new singleton for consistency checker.

* Recruit the consistency checker only if enabled.

* Add a yield in monitorConsistencyChecker().

* Minor fixes.

* Consistency check workload enhancements.

* Minor fixes and clarifications.

* clang format

* Clang format.

* Minor fixes, cleanup, debug tracing.

* Misc.

* Move the consistency scan information from dbconfig to a key backed object.

* Move consistency scan config out of db cofig to a state object and feature rename.

* ConsistencyCheck workload refactor.

* devFormat

* Update fdbcli/ConsistencyScanCommand.actor.cpp

* Review Comments.

Co-authored-by: negoyal <neelam.goyal@gmail.com>
Co-authored-by: Ata E Husain Bohra <ata.husain@snowflake.com>
2022-09-16 09:03:06 -07:00
Dennis Zhou e6222be6c6 fdbcli: use ApiVersion::LATEST_VERSION instead of hardcoding 2022-09-12 17:08:57 -07:00
A.J. Beamon 726d5215a0
Remove API 720 guards for tenants (experimental feature) and the cluster ID special keys (#8108)
* Remove API 720 guards for tenants (experimental feature) and the cluster ID special keys (no need to guard)

* Enable the relaxed special key access in transactions that need to use special key-space APIs introduced in 7.2
2022-09-08 17:22:36 +02:00
Junhyun Shim bcf0861a0a Make fdbcli check for TLSConfig and coordinator address mismatch
At least one of the coordinator addresses in cluster file must contain ":tls" suffix
if fdbcli's resolved TLS client configuration holds any of the TLS elements (key, cert, or CA)

Conversely, if none of the TLS elements are configured,
at least one of coordinator addresses must be without ":tls" suffix
2022-09-01 15:50:11 +02:00
Chaoguang Lin 3fed0456ca
Add an verify option for \xff\xff/worker_interfaces special keys (#7873)
* Add the verify option for \xff\xff/worker_interfaces

* Remove unused code

* update documentations

* update documentations

* solve comments from review

* update some of the comments to be more clear
2022-08-15 14:05:07 -04:00
Vaidas Gasiunas 79571dd2b4
Testing upgrades to a future version of FDB (#7780)
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli

* Auto format python files used in upgrade tests

* Add a test for upgrading to a future FDB version

* Emphasize that the options for using future protocol version are intended for test purposes only

* Make the global variable for current protocol version visible only locally

* Refactirng to avoid using currentProtocolVersion() in static intialization

* Update go bindings
2022-08-08 17:29:49 +02:00
A.J. Beamon 7f53b340f5 Merge branch 'main' into feature-metacluster
# Conflicts:
#	bindings/python/tests/fdbcli_tests.py
#	fdbcli/TenantCommands.actor.cpp
#	fdbclient/ClientKnobs.cpp
#	fdbclient/include/fdbclient/ClientKnobs.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbserver/BlobManager.actor.cpp
#	fdbserver/BlobWorker.actor.cpp
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
#	flow/include/flow/error_definitions.h
2022-07-28 11:43:21 -07:00
A.J. Beamon 7c6b3fb0b8 Merge branch 'main' into feature-metacluster 2022-07-27 08:55:10 -07:00
Jon Fu 6dbaf46ec8 restructure test workload impl and map 2022-07-26 13:53:04 -07:00
A.J. Beamon f2a8af9155 Better API version handling for fdbcli tenant commands 2022-07-26 09:04:29 -07:00
A.J. Beamon a64693518a Add support for tenant groups 2022-07-26 09:04:29 -07:00
A.J. Beamon 279296c29f Merge branch 'tenant-metadata-change' into feature-metacluster
# Conflicts:
#	fdbclient/SystemData.cpp
#	fdbclient/Tenant.cpp
#	fdbclient/include/fdbclient/SystemData.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbclient/include/fdbclient/TenantManagement.actor.h
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-20 09:18:27 -07:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
A.J. Beamon 860d3843cc Merge remote-tracking branch 'origin/feature-tenant-groups' into feature-metacluster 2022-07-16 19:33:26 -07:00
A.J. Beamon c08592368f Add support for tenant groups 2022-07-15 21:18:41 -07:00
A.J. Beamon 0fe6f6ad36 Merge remote-tracking branch 'origin/update-prefix-encoding-in-tenant-metadata-json' into feature-metacluster
# Conflicts:
#	bindings/python/tests/fdbcli_tests.py
#	fdbcli/TenantCommands.actor.cpp
#	fdbclient/include/fdbclient/TenantManagement.actor.h
#	fdbclient/include/fdbclient/TenantSpecialKeys.actor.h
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-14 14:32:29 -07:00
A.J. Beamon cb499fa5db Merge branch 'main' into feature-metacluster 2022-07-13 15:28:34 -07:00
A.J. Beamon 91ccdbcb03 Use a more standard encoding in the JSON metadata reported for tenants. 2022-07-13 15:22:19 -07:00
A.J. Beamon a3f970cf3a Parse integer options in fdbcli rather than passing the bytes unparsed. This allows specifing the string "1" instead of "\x01\x00\x00\x00\x00\x00\x00\x00". 2022-07-11 16:39:59 -07:00
A.J. Beamon b9cc5389b1 Undo some changes that aren't needed 2022-07-07 14:48:36 -07:00
A.J. Beamon a32cc5218c Fix some merge issues 2022-07-07 11:55:31 -07:00
A.J. Beamon 1519f24f77 Merge branch 'main' into feature-metacluster 2022-07-07 09:35:40 -07:00
A.J. Beamon aea4d802c6 Merge branch 'main' into feature-metacluster 2022-06-30 15:08:09 -07:00
Jon Fu 5e7bb0aa21 Merge branch 'main' of github.com:apple/foundationdb into jfu-tenant-rename 2022-06-29 13:20:19 -07:00
sfc-gh-tclinkenbeard 2134fa9a0c Merge remote-tracking branch 'origin/main' into global-tag-throttling3 2022-06-28 19:51:41 -07:00
A.J. Beamon e1a93988ef Merge branch 'main' into feature-metacluster 2022-06-28 14:58:07 -07:00
Jon Fu 869c6fcac5 reintroduce renameTenantCommandActor as a pass-through function 2022-06-28 11:35:40 -07:00
Jon Fu 904ac358ce call directly into management api and remove extra function 2022-06-28 11:35:38 -07:00
Jon Fu ddfdbdcb09 initial commit introducing renameTenant function 2022-06-28 11:34:44 -07:00
sfc-gh-tclinkenbeard 086e4bff06 Merge remote-tracking branch 'origin/main' into global-tag-throttling3 2022-06-28 10:18:13 -07:00
Markus Pilman fc2f695b15 Merge remote-tracking branch 'origin/main' into features/cmake-refactor-dirs 2022-06-28 10:41:07 -06:00