Commit Graph

454 Commits

Author SHA1 Message Date
Johannes Scheuermann 5ce16e8e1e
DD maintenance duration observability (#13767)
* Add maintenance mode duration information to trace events

* Persist start time to provide the correct duration if DD gets restarted or re-recruited
2026-07-31 19:33:05 +02:00
Trevor Clinkenbeard 3dff995742 Merge remote-tracking branch 'origin/main' into dev/tclinkenbeard/native-cdc-reverse-current-tag 2026-07-11 06:24:49 -07:00
Michael Stack c4c1be3f41
fdbcli audit_storage metadata_encoding: fix counting of format-neutral entries (#13670)
* fdbcli audit_storage metadata_encoding: fix counting of format-neutral entries

The audit tool's status line reported MIGRATION IN PROGRESS
indefinitely on any running cluster in the FORWARD direction — the
FORWARD COMPLETE terminal state was unreachable. Root cause: two
categories of format-neutral entries were being counted as old-format:

1. Empty-value keyServers entries. These are KRM boundary sentinels
   (see krmDecodeRanges in fdbclient/KeyRangeMap.cpp:70-84 for how
   they are emitted at range ends). They mark the edge between
   adjacent same-valued ranges and do not carry any keyServers
   assignment. Every cluster has some — they cannot be migrated away.

2. serverKeysFalse ("this server does not own this range"). Written
   by both finishMoveKeys (old format, MoveKeys.cpp:1787) AND
   finishMoveShards (new format, MoveKeys.cpp:2138) on the drop-side
   of any move. Also written as range boundary markers around every
   assigned range (see unassignServerKeys in MoveKeys.cpp:91,127).
   Every running cluster has many — they cannot be migrated away.

Under the old counting logic, keyServersOld and serverKeysOld were
therefore never zero, and the terminal-state condition
(keyServersOld == 0 && serverKeysOld == 0) that gates FORWARD COMPLETE
never triggered. ROLLBACK COMPLETE was unaffected because its
condition looks at *New* counts.

Fix: exclude format-neutral entries from the counts. keyServers skips
empty values. serverKeys uses two new small helpers added to
SystemData.h/cpp:

  isServerKeysUnassigned(value)      — true for empty + serverKeysFalse
  isServerKeysOldFormatAssigned(value) — true for serverKeysTrue +
                                         serverKeysTrueEmptyRange

Not caught earlier because the audit command has no automated
callers — it is a manual operator command, and manual users would
see MIGRATION IN PROGRESS and assume DD needed more time. The first
automated consumer that asserts the FORWARD terminal state is the
k8s test test_shardencode_rollover_load, which hangs indefinitely
without this fix.

* audit_storage metadata_encoding: workload counterpart + review fixes

Round out PR1 with the sim-workload half of the counting fix plus four
fixes surfaced by code review of the audit-tool changes:

1. Sim workload counting fix. fdbserver/workloads/CheckMetadataEncoding.cpp
   had the identical miscounting bug as the fdbcli command — counted KRM
   boundary sentinels as keyServersOld and serverKeysFalse as
   serverKeysOld. Now uses the same isServerKeysUnassigned /
   isServerKeysOldFormatAssigned classifiers introduced in this PR.

2. Wrap the fdbcli dataMoves scan in a retry loop. Previously a
   retryable getRange error would swallow via tr.onError() without
   re-reading, leaving dataMovesCount at 0 and causing a false
   "ROLLBACK COMPLETE — safe to downgrade binary" report while data
   moves were still in flight.

3. Move terminal-state assertions (requireForwardComplete /
   requireRollbackComplete) out of the shardEncodeExpected branches.
   Previously the assertions were nested inside the branch that matched
   the knob, so a TOML that requested the assertion while the knob
   override failed to take effect would silently skip the check. Now
   the assertions always run when requested, and mismatched
   knob/option combinations fail loudly with an explicit
   misconfiguration error.

4. Emit the DataMoves trace detail only when the dataMoves range was
   actually scanned. Previously the trace unconditionally logged
   DataMoves=0 even when no scan ran (requireRollbackComplete=false),
   poisoning downstream log parsers that assume 0 means "no data
   moves in flight" rather than "not measured".

Also adds tests/fast/CheckMetadataEncodingForward.toml requiring
requireForwardComplete=true — the direct regression test for the
counting fix (without the fix, KRM sentinels/serverKeysFalse are
miscounted as old-format and FORWARD COMPLETE is unreachable).
2026-07-10 11:02:14 -07:00
Trevor Clinkenbeard b70de5d319 Read latest Native CDC tag directly 2026-07-09 15:57:20 -07:00
Trevor Clinkenbeard 93cd3f873c Fix Native CDC clang-tidy warnings 2026-07-09 10:42:31 -07:00
Trevor Clinkenbeard cbb0b5725e Merge remote-tracking branch 'origin/main' into dev/tclinkenbeard/pr-13287-review-followup
# Conflicts:
#	fdbclient/include/fdbclient/FDBTypes.h
#	fdbserver/core/include/fdbserver/core/WorkerInterface.actor.h
#	flow/ProtocolVersion.h.cmake
#	flow/ProtocolVersions.cmake
2026-07-06 15:32:57 -07:00
Akanksha Mahajan f6a0557f3a
Rename Range Partitioned to consisent name (#13386) 2026-07-03 11:02:24 -07:00
Trevor Clinkenbeard 7267563325 Fix SSI serialization unit test outside simulation 2026-06-25 13:42:51 -07:00
Trevor Clinkenbeard ea3e4c3196 Use comparison-specific assertions for native CDC 2026-06-04 16:53:22 -07:00
Trevor Clinkenbeard e31cb45c22 Improve type safety for tag history 2026-06-04 15:34:37 -07:00
Trevor Clinkenbeard ecb82cd453 Merge remote-tracking branch 'origin/main' into dev/tclinkenbeard/native-fdb-cdc 2026-06-03 19:32:31 -07:00
Akanksha Mahajan 3c247e928a
Coordinate BackupAgent and DataDistributor to compute partitions for range-partitioned backup (V3) (#13304)
* Implementation

* Add knob

* Addressed comments

* Addressed comments to move clearing the key in checkAndDisableRangeBackupWorkers
2026-06-03 08:30:26 -07:00
Akanksha Mahajan 7c36482ad4
Persist partition map history to system keys for recovery (#13283)
* Persist and read partition map

* Addressed comments

* Addressed comments
2026-06-01 13:58:41 -07:00
Trevor Clinkenbeard 270b3d700f Fix CDC lifecycle recovery, retired-tag cleanup, and expiry handling 2026-05-27 13:27:33 -07:00
Trevor Clinkenbeard 1fce475ae7 Enable unit tests in simulation 2026-05-27 11:35:28 -07:00
Trevor Clinkenbeard fc075ffbe8 Move native CDC acknowledgements to storage-backed system keys 2026-05-27 10:19:24 -07:00
Trevor Clinkenbeard 7ada95e40b Anchor CDC retention at stream registration commit versions 2026-05-27 05:57:28 -07:00
Trevor Clinkenbeard 1808150623 Bound CDC tags and persist stream ID allocation 2026-05-27 05:57:28 -07:00
Trevor Clinkenbeard 92752c5a64 Add buffered native CDC consumption and safe shared-tag acknowledgements 2026-05-27 05:57:27 -07:00
Trevor Clinkenbeard 9f52b0cae7 Persist and publish CDC stream ownership routing 2026-05-27 05:57:27 -07:00
Trevor Clinkenbeard 9ce0954a67 Add native CDC stream lifecycle metadata operations and tests 2026-05-27 05:57:27 -07:00
Trevor Clinkenbeard 158333e123 Add native CDC commit proxy routing 2026-05-27 05:57:26 -07:00
Trevor Clinkenbeard d86ffe0bf3 Initial native CDC commit 2026-05-27 05:57:26 -07:00
Akanksha Mahajan 5518f8b6ac
Implement SetBackupKey and Other remaining functions (#13144)
* Implement SetBackupKey and Other remaining functions

* Kept single keys

* Addressed comments
2026-05-13 14:30:59 -07:00
Trevor Clinkenbeard ed576c36d8
Merge pull request #13195 from tclinkenbeard-oai/dev/tclinkenbeard/key-backed-header-rename
Rename key-backed coroutine headers
2026-05-11 09:11:04 -07:00
neethuhaneesha 2633ed5793
Adding range_backup_worker_enabled to DB config for recruiting range partitioned backup workers. (#13169) 2026-05-11 08:32:09 -07:00
Trevor Clinkenbeard 287efc8530 Rename key-backed coroutine headers 2026-05-11 07:47:09 -07:00
gxglass 2f0158b4c2
Remove dead code left behind by blob worker, change feed, and metacluster feature deletions (#13119)
Prior efforts (PR#12435, PR#12470, PR#12486, PR#12583, PR#12667, PR#12903) removed some experimental features. This PR cleans up some leftover pieces.

I suppose this is basically a matter of historical interest at this point but those PRs were done purely by hand (ok, with like grep and a text editor) i.e. without any AI coding assistance. So it is not surprising that there is a few percent of leftover bits here and there.

Fixes:

Remove monitorBlobWorkers() and blobRestoreCommandActor() declarations that had no implementations (linker bombs if called)
Remove fast_restore from setclass help text (previously directed operators into an ASSERT(false) crash)
Dead knob removal (13 knobs):

12 blob worker ratekeeper knobs (BW_THROTTLING_ENABLED, TARGET_BW_LAG, etc.) and BLOB_WORKER_PAGE_CACHE
Remove bwLagTarget field from RatekeeperLimits and associated plumbing in Ratekeeper
Dead code removal:

4 never-incremented storage server counters (feedBytesFetched, changeFeedMutations, changeFeedMutationsDurable, changeFeedDiskReads)
Assigned-but-never-read nonExpanded variable in StorageServer::addMutation
ClusterNameRef/ClusterName typedefs, RestoreLoader/Applier/Master forward declarations
Dead schema strings (blob_worker_lag, blob_worker_missing, unreachable_blobManager_worker, metacluster_metrics_missing)
Dead cacheKeys*/cacheChange* function implementations in SystemData.cpp
Dead tenant group code in FuzzApiCorrectness.cpp
Dead fdbcli constants (msgClusterTypeKey, msgDataClustersKey) and blobrange history filter
Dead file removal:

contrib/mockkms/ -- unused Go mock KMS server
tests/fast/EncryptionUnitTests.toml -- tests a path (/blobCipher) that no longer exists
2026-04-29 20:15:54 -07:00
Trevor Clinkenbeard e38baf18ea Fix more clang-tidy warnings 2026-04-15 14:23:33 +00:00
Akanksha Mahajan 77622b9000
Monitor and Update Progress of BackupWorkers (#12844)
* Monitor and Update Progress

* Implement BackupRagePartitionedProgress as needed.

* Move the function to the end

* Complete implementation

* Addressed comments
2026-04-10 13:06:15 -07:00
Trevor Clinkenbeard 6f86402aae
Merge pull request #12923 from tclinkenbeard-oai/dev/tclinkenbeard/tdmetric-coro 2026-04-05 10:36:45 -07:00
Trevor Clinkenbeard adad276a70 Rename TDMetric.actor.h 2026-04-02 05:51:35 +00:00
gxglass bca167fe96
Remove parallel restore feature (#12903)
This parallel restore feature has been slated for removal for at least a year. This PR is closely based on earlier PR #12107.

This blog post explains some of the problems with the parallel restore feature: https://medium.com/@jingyuzhou/why-foundationdb-restore-is-slow-and-what-can-be-done-about-it-e73a821fdd33

As far as large feature removal changes go, this one is very straightforward, with most relevant files and test cases simply being deleted. There is one knob rename where storageserver.actor.cpp was using a knob with FASTRESTORE in the name. Other than that, changes to shared files mainly involve removing fastrestore-specific CLI and role support.

In progress:
20260330-222511-gglass-5ee0142213471b70 compressed=True data_size=35343375 duration=4611964 ended=100000 fail=1 fail_fast=1000 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=0:58:23 sanity=False started=100000 stopped=20260330-232334 submitted=20260330-222511 timeout=5400 username=gglass

The one failure was in SwizzledCycleTest.toml with too many lines of output and a timeout. I kind of suspect it's unrelated but haven't looked further.
* Remove parallel restore feature.  This is based on earlier PR 12107.  Compiles but untested.

* AI generated commit:

⏺ The fix restores a single if block that was accidentally deleted when removing the FASTRESTORE_TOOL code:

  if (!restoreSystemKeys && !restoreUserKeys && backupKeys.empty()) {
      addDefaultBackupRanges(backupKeys);
  }

  When no explicit key ranges are specified on the command line and neither --user-data nor --system-metadata flags are set, this populates backupKeys with the default backup ranges
  (essentially all user data). Without it, backupKeys stays empty and hits the ASSERT(!backupRanges.empty()) in submitBackup().

* Remove a believed-to-be-dead code path, and update .gitignore

* Remove duplicate definition of restoreRequestDoneKey
2026-04-01 15:57:12 -07:00
Akanksha Mahajan 5cce30683b
Implement UploadData and Progress API for Backup V3 (#12811)
* Implement UploadData API and Progress under a key

* Rebase and Update code
2026-03-24 00:11:47 -07:00
Trevor Clinkenbeard 72dd0bfd2a Move GlobalConfig.actor.h into GlobalConfig.h 2026-03-21 01:02:41 +00:00
Michael Stack 1f75fbe25b
Improve backup/restore observability and status messages (#12729)
* Add BulkDump/BulkLoad observability for backup/restore progress tracking

Progress tracking APIs:
- getBulkDumpProgress() / getBulkLoadProgress() - track tasks, bytes, throughput, ETA
- getBulkDumpOwner() - track job ownership (e.g., which backup owns a bulkdump)
- Owner registration system for backup agents

CLI improvements:
- 'bulkdump status' and 'bulkload status' commands with human-readable output
- Progress display with throughput and ETA

Backup agent integration:
- Register backup as owner of bulkdump jobs
- Show bulkdump/bulkload progress in backup status output

Utility functions (flow/Util.h):
- formatBytesHumanReadable() - "1.23 TB", "456 MB"
- formatDurationHumanReadable() - "2 hours 30 minutes"
- formatBytesProgress(), formatThroughputLine(), formatETALine()

Test coverage:
- verifyBulkDumpObservability() / verifyBulkLoadObservability() in BackupS3BlobCorrectness

Bug fixes:
- Fix format string mismatch crash in FileBackupAgent progress display
- Add empty result guards for krmGetRanges() calls to prevent size_t underflow

* Formatting

* Rebase and formatting

* * cmake/CompileRocksDB.cmake
 Preserve version when find_packge doesn't turn up rocksdb version

* fdbbackup/tests/backup_tests_common.sh
 Wait when mode is 'both' for snapshot to be written.

* fdbclient/BulkLoading.cpp
 Remove extra '/'.

* Output time of bulkload in bulkload history fdcli output as iso8601 instead of seconds since epoch

* Print start time as iso8601 -- readable -- instead of seconds since epoch -- unreadable... And don't crash if not a UUID

* Remove cute graphics.. and bullet characters

* Throw exception if not able to parse backup 'mode'
2026-03-18 21:10:28 -07:00
Trevor Clinkenbeard 652f85de07
Add and enforce more `readability-*` `clang-tidy` rules (#12765)
* Add and enforce more readability-* clang-tidy rules

* Revert readability-else-after-return changes
2026-03-12 22:30:38 -07:00
Akanksha Mahajan 789eed6ce7
Pull, pop and upload PartitionMap from TLOG for Backup v3 (#12718)
* Peek and Upload Partition Map

* Fix error and add additional comments

* Addressed comments
2026-03-08 17:45:00 -07:00
Trevor Clinkenbeard a135fb34ff
Use structured bindings in for loops (#12744)
* Use structured bindings in for loops

* Simplify more for loops
2026-03-03 18:48:55 -08:00
neethuhaneesha 1e6a69fcd1
Removed no-op mode in BackupWorker (#12746) 2026-03-03 15:11:46 -08:00
Trevor Clinkenbeard fafbfb4641
Remove dynamic knobs feature (#12683)
* Remove dynamic knobs feature

* Add back deprecated command line arguments

* Remove dead TransType enum

* Mark configuration database feature deleted

* Remove deprecated encryptModes from test files

* Remove outdated comment

* Remove duplicate feature line in feature-status.md

* Address compilation warnings
2026-02-19 13:25:00 -08:00
Dan Lambright 14496da6aa
Remove generate synthetic data feature; superceeded by bulk load (#12707)
Co-authored-by: Dan Lambright <hlambright@apple.com>
2026-02-14 15:52:12 -08:00
gxglass c62bb6bf38
Delete encryption at rest (#12667)
Delete encryption at rest in accordance with plans circulated in PR #12400.

Development of this PR was mostly straightforward. Several issues of note:

Upgrade tests which use Redwood want to use the old "encrypt by 0xFF XOR" test-only encoding/encryption algorithm. I wanted to delete that in main and did so. Upgrade tests have been modified not to use Redwood (storage engine 3, mentioned in various storageEngineExcludeTypes test options updates). 7.1 did not define storageEngineExcludeTypes. After some discussion we have decided to delete 7.1-based upgrade tests. 7.3- and 7.4-based upgrade tests remain in place.

Cross-version compatibility (such as it is, I assume in client library startup) remains super easy to break when deleting code. Tips: (A) removing unnecessary arguments to functions is bread and butter code editing when removing code, but if you find yourself removing arguments from a serializer() invocation, you might be breaking a protocol. (B) run ctest -R upgrade early and often.

An implication of these changes is that we are going to rely on the community to make us aware of any {upgrade+Redwood}-specific bugs. In other words, we are declining to continue to go out of our way to test functionality that we have no plans to use.

On the plus side this reclaims 15,000 more lines of code that we don't have to look at or think about, including in common areas such as commit proxy, backups, and generic storage server code.

Testing:
20260129-230241-gglass-15694f5f80af6932 compressed=True data_size=34905446 duration=4335357 ended=100000 fail_fast=1000 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=6:21:47 sanity=False started=100000 stopped=20260130-052428 submitted=20260129-230241 timeout=5400 username=gglass

20260202-214159-gglass-69b90c779cf8ec68 compressed=True data_size=35007141 duration=4612639 ended=100000 fail_fast=1000 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=3:00:25 sanity=False started=100000 stopped=20260203-004224 submitted=20260202-214159 timeout=5400 username=gglass

* Checkpoint file removals and code edits for removing encryption at rest. Have not tried to compile this yet.

* Checkpoint some improvements.  Still does not compile.

* Merging with upstream changes in copyright updates resurrected previously deleted files, so delete them again

* Checkpoint incremental progress towards getting this to compile

* Checkpoint more intermediate changes getting encryption at rest deleted.  Still doesnt compile but getting closer.

* Changes sufficient to get things to compile with removal of encryption at rest.  NOT TESTED.

* Delete encryptModes from toml files run by current fdbserver binaries; restarting tests using <= 7.4 binaries do need encryptModes to say disabled

* Avoid using MAX_ENCODING_VALUE for random purposes for which other solutions are more clear and generally better

* Stop using XOREncryption_TestOnly because that no longer works.  Also I see no need to support it as it requires unneeded interfaces and test fixtures in order to actually work.

* Remove more page encryption stuff, and address some TODO(gglass) comments.

* debugging redwood failures

* Fix some misc simulation failures.  Notably, disable storage engine type 3 (redwood) on upgrade tests, as it writes databases with encoding type 1 which is no longer supported

* Add a comment discussion deprecation options for fields in persistent metadata, and explain why we are merely renaming the member as deprecated and nothing anything else.

* Remove 7.1 upgrade tests.  These tests enable Redwood and write databases with "0xFF XOR encryption" style encoding (encoding 1), which is removed in main.

* Improve comments

* Obligatory f3f commit.  Format The Effin Source Files

* Delete more unneeded encryption stuff

* Put back about 1% of deleted code in a desperate attempt to unbreak broken protocol compatibility

* formatting

* Remove mentions of encryption at rest in backup related APIs

* Address misc review comments.  Remove --encrypt-files backup option.
2026-02-04 16:02:06 -08:00
neethuhaneesha 4e236b0285
Fix for toggling backup_worker_enabled during backup v2 causing assert. (#12666) 2026-02-03 12:06:13 -08:00
Jingyu Zhou 2d2a2144f4
Update copyright years to 2013-2026 (#12653)
No functional changes.
2026-01-22 10:49:41 -08:00
Michael Stack aa35d6cc29
Add restore validation feature: restores to special keyspace allowing validating backup/restore in single cluster (space willing) (#12573)
* Add restore validation feature with simplified backup gap fix

Implements restore validation using audit_storage to verify backup/restore
correctness. Includes a minimal fix for the backup gap bug.

Key components:
- ValidateRestore audit type: compares source keys against restored keys
  at \xff\x02/rlog/ prefix in storage server
- DD audit fixes: propagate validation errors, handle DD failover correctly
- RestoreValidation and BackupAndRestoreValidation workloads for testing
- Simplified backup gap fix: prevent snapshot from finishing in the same
  iteration it dispatches the last tasks (single flag + one check)
2026-01-07 15:23:02 -08:00
gxglass bab7637d87
Delete multitenant and metacluster features (#12583)
These features have been previously marked for deletion per PR #12400.

This change necessarily affects a lot of files. In general I found it preferable to cut along the FDB <-> tenant boundary, rather than try to cut tenant into multiple pieces, stitch the Frankenstein tenant implementation back together with FDB, and generally remove the limbs one by one. So it is a single big deletion.

Note that some tenant-related metadata has been written in a non-flag-controlled manner by prior releases and probably must be ignored indefinitely. Fortunately this is isolated to include/fdbclient/ClientLogEvents.h. (Details: deleting an Optional from a serialized struct results in deserialization of garbage in upgrade tests. The serialized nullopt to indicate "no Tenant" is formally part of FDB persistent metadata even in FDB clusters that never would have enabled the tenant feature.)

During the course of testing these changes, many interesting bugs were encountered. I won't discuss details of them here. Causes range from flat out damage (by me) to production code in the course of removing tenant related bits (mainly in NativeAPI.actor.cpp and CommitProxy.actor.cpp), damage to various workload files (particularly FuzzApiCorrectness.actor.cpp, which is very sensitive to changes), and many toml files needing updated test flags/options.

More testing details: https://quip-apple.com/Zr6VAycxoli9

20251209-012852-gglass-8ff850b772d868f2 compressed=True data_size=35311687 duration=21671404 ended=500000 fail_fast=1000 max_runs=500000 pass=500000 priority=100 remaining=0 runtime=2:31:30 sanity=False started=500000 stopped=20251209-040022 submitted=20251209-012852 timeout=5400 username=gglass

* remove some unneeded tests, and remove mentions of deleted tests from tests/CmakeLists.txt

* Initiate removal of metacluster. NOTE: this seems to also want removal of tenant. Consider removing them together.

* work on removing metacluster

* delete files with `Tenant` in the name, having reviewed them to ensure that they basically contain what the name implies

* fdb_c.h: remove prototypes for C API methods which have been deleted (blob granule) or which are so long deprecated that they are outside any reasonable/documented support window

* Surgical removal of tenant references from files in bindings/ top level directory.  Compilation not yet attempted.

* Surgical removal of tenant related stuff from fdbcli/ top level directory.  Compilation not yet attempted.

* Misc tenant code removal, and other stuff which I think may not be needed.  Compilation still not attempted.

* Remove more tenant or tenant-adjacent or blob-granule-adjacent stuff.  Or at least stuff that looks adjacent to that stuff.  Not compiled or tested.

* Start removing Tenant stuff from fdbclient/.  Far from complete.  Compilation not attempted.

* Remove tenant references from many source files.  There are still about 7 principal fdbclient/ and fdbserver/ files with a lot of tenant logic left to delete. Also, all of fdbserver/workloads needs to be looked at.  Still have not attempted compilation.

* Remove tenant entanglement from watch functionality

* Remove tenant stuff from fdbserver/tester.actor.cpp

* Delete metacluster workloads

* Remove tenant related stuff from workloads.  Also taken the liberty of removing some functionality that appears unused or untestable by Apple.

* Checkpoint tenant removal from FuzzApiCorrectness.actor.cpp

* NativeAPI.actor.cpp: `Tenant` has left the building.

* SimulatedCluster.actor.cpp: `Tenant` has left the building

* DDShardTracker.actor.cpp: Tenant evicted

* storageserver.actor.cpp: `tenant` has left the building.

* fdbserver/workloads/FuzzApiCorrectness.actor.cpp: remove tenant references, but some lingering cleanup needed in `loadAndRun`

* FileBackupAgent.actor.cpp: tenant has left the building

* CommitProxyServer.actor.cpp: remove tenant

* Remove more tenant references from misc files such as bindings tests, documentation, and some fdbserver headers I left earlier

* Fix missing-file errors in CMakeLists.txt files.  This is the first attempt to compile this stuff.

* checkpoint misc changes to fix compile errors

* checkpoint more compile fixes

* StorageServerInterface.h: put back more verify() calls

* More misc compile fixes

* whole bunch of misc fixups including some code put-backs to address compile errors

* More compile fixes

* More compile fixes.  Still does not compile.

* incremental compile fixing

* ...

* ...

* Checkpoint a bunch of compile fixes.  Not quite there but getting closer

* More compile fixes.  There seem to be about 10 files left, mainly CommitProxyServer.actor.cpp and storageserver.actor.cpp

* IT COMPILES NOW.  THIS IS STILL ALL UNTESTED.  Unsurprisingly, CommitProxyServer.actor.cpp and storageserver.actor.cpp took the most tweaking.

The updates in CMakeLists.txt and workloads/UnitTests.actor.cpp are basically trivial and mainly reflect
the ordering of dependencies -- that stuff didn't get attempted until all of fdbserver compiled.

* Put back one block relating to encryption at rest mode.  Simplify some TODO(gglass) instances.

* Put back some encryption related knobs

* remove `enable_tenants` from local_cluster.py to maybe fix some ctests

* Remove tenant related options from toml files.

* feature-status.md: add a line for encryption at rest, which seems to have been added for multi-tenant; status is now in doubt

* Fix a pretty bad bug introduced in tenant deletion; ensure we dont attempt to construct a std::string of negative length

* workloads/FuzzApiCorrectness.actor.cpp: avoid division by zero

* flow/Platform.actor.cpp: add a try/catch wrapper around side threads; emit a better addr2line type command

* NativeAPI.actor.cpp: fix a bug introduced in tenant removal relating to reporting conflicting keys under conflictingKeysRange

* ReportConflictingKeys.actor.cpp: separate an ANDed assert into two asserts

* SpecialKeySPaceCorrectness.actor.cpp: put back some logic removed with tenant removal.  This test was failing due to a bug with conflict key range reporting.  Fixed separately in NativeAPI.actor.cpp.

* remove QuotaCommand.actor.cpp

* Force disable tenant and encryption on disk in upgrade tests

* Add back file I guess I deleted?  who knows

* put back another file

* design/feature-status.md: update the new row for encryption at rest to firm up the claim that it is experimental, unowned, and scheduled for deletion

* Remove EncryptKeyProxyTest since we do not use it

* new file tests/slow/BulkDumpingS3WithChaos.toml: remove tenantModes setting

* Undo damage to pushToBackupMutations() from removing tenant feature.  This caused inverted_range errors and failed commits in backup related simulations.

* tests/restarting/from_7.4.0/Snap*-1: ensure that tenantModes = disabled

* Try again on workloads/FuzzApiCorrectness.actor.cpp

* simplify tenant-free (mostly) FuzzApiCorrectness workload code

* try harder to remove lingering tenant-related brokenness from FuzzApiCorrectness.actor.cpp

* Explicitly specify tenantModes = ['disabled'] in all the -1 restart files

* Remove tenantModes from 7.1-based upgrade tests as its an unknown option.  Hopefully the code doesnt actually turn on tenant stuff

* do not specify tenantModes in downgrade tests

* Downgrade test to_7.4.5: dont say tenantModes

* more tenantModes updates

* Remove a legacy allowDefaultTenant that no longer is meaningful in downgrade to 8.0

* Put back empty Optional<TenantName> turdlets into serialized log events to avoid breaking ClientTransactionProfilingCorrectness upgrade tests (even with tenantMode = disabled)

* disable encryption on a few more upgrade related test cases.  That feature is slated for removal anyway

* Remove unneeded workload files that have been subject to #if 0 for a while. Remove commented out block in ClusterRecovery

* disable encryption in more upgrade tests

* Remove choice four-letter words from commentary

* Format 42 files

* Try to fix a doc bug failing the CI build

* More doc compilation error fixes

* Delete more tenant junk from documentation

* fix spelling mistake in comment

* Remove deleted cross-references from documentation.  This necessitated editing release 3.0.0 release notes, which is insane.

* Remove more tenant stuff from bindings tests

* Remove more tenant bits from design/ files

* Remove more tenant related stuff

* Delete more tenant references.  Put back ten-ant spellings as tenant now that grep output is substantially reduced.

* Put back some tenant stuff into apitester; its deletion seems to have introduced bugs.  Also whine about comments some more, because, really, the comments deserve it.

* Updates to workload files and one other thing based on review comments

* de-actorify decodeKVPairs

* format one source file

* Restore transaction tagging doc

* Restore throttle doc details in administration.rst

* Restore fdbserver/workloads/GetEstimatedRangeSize.actor.cpp and associated toml file, minus tenant stuff

* bindings/c/test/{shim related}: update comments and disable functionality that no longer works post-tenant

* put the cli-throttle tag back in

* bindingtester: fix python syntax errors

* remove useless comment

* Remove comment about useless comments, and remove the useless comments
2025-12-09 12:39:41 -08:00
gxglass 668450f22e
Delete Storage Cache Server feature (#12486)
* Initiate deletion of storage cache feature.  This is rough and is mostly done by commenting out code in case backtracking is needed.  Compiles.  Not tested.

* fix some test errors about cache consistency check options which we no longer care about

* design/feature-status.md: Storage Cache status updated to `has been deleted`.

* Delete it for real

* disable BackupS3BlobCorrectness.toml because it fails a lot

* In the interest of a single-purpose, clean diff, put back a removed dumb warning that generates compile error noise

* fix formatting

* Add TODO comment to remove tagLocalityLogRouter

* fix typo
2025-10-21 15:04:39 -07:00
gxglass f7913c121d
Delete ChangeFeed feature (#12470)
Continuing the deletions of unowned experimental features as listed in #12400.

ChangeFeed is mostly contained in NativeAPI.actor.cpp and storageserver.actor.cpp, with a modest amount of code in dedicated files and a scattering of updates in misc other places where features tend to pile into.

There are a few lingering TODOs for fine tuning of the additional removal, including a state machine in storageserver.actor.cpp. My preference is to checkpoint this diff before continuing with more experimental/risky fine grain surgery in close proximity to code which must remain. This PR nets -6000 lines, mostly in NativeAPI.actor.cpp and storageserver.actor.cpp, so benefits should accrue in terms of compile times and general "less unwanted code showing up on your screen" when working in these files.

Ran overnight:
20251016-003236-gglass-46416fec30cddcb0 compressed=True data_size=38462735 duration=14586409 ended=310182 fail=10 fail_fast=10 max_runs=500000 pass=310172 priority=100 remaining=0 runtime=1:21:30 sanity=False started=314133 stopped=20251016-015406 submitted=20251016-003236 timeout=5400 username=gglass

The 10 failures were in a specific recent unit test failure not related to this PR (link to details shared in Slack).

Prior to this:
20251015-232157-gglass-46416fec30cddcb0 compressed=True data_size=38462735 duration=5241939 ended=99998 fail=2 fail_fast=10 max_runs=100000 pass=99996 priority=100 remaining=0 runtime=1:10:39 sanity=False started=100000 stopped=20251016-003236 submitted=20251015-232157 timeout=5400 username=gglass

Those 2 failures were one existing bug (there is a radar for it) and one where amusingly Joshua decided to run a deleted BlobGranule test case, which I am just going to ignore.


* Initiate deletion of changefeed feature.  Probably does not compile

* Checkpoint removal of changefeed reature.  This set of changes compiles but is untested.

* feature-status.md: ChangeFeed: status is now `has been deleted`.

* Format code.  This passed 100k simulations minus 2: one was, Joshua running a deleted test case (wtf?) and the second was a test case with an open radar
2025-10-16 22:49:10 -07:00
gxglass b1d6dcf0e7
Delete blob granule feature (#12435)
This is the first experimental feature to be deleted in the list published at PR #12400.

There is more code here than I anticipated. It is about 40,000 lines total, of which about three quarters are in dedicated files which I am deleting, and about one quarter is in shared files. That means about 10k lines in shared files, which is the stuff we tend to notice day to day (that plus the test failures on heretofore not-yet-disabled test cases, which I am now deleting).

I ran 3 million simulations, mostly against 692df86 or very similar code (differing by one TraceEvent). This was prior to syncing with upstream/main, which had no conflicts and from which I don't expect problems. The number of failures in these runs was about 8. We looked at them and believe there is a high likelihood that these are existing issues not related to the changes in this PR. More details on these failures can be found in docs linked from here: https://quip-apple.com/MN7gAyXLjgyn

* change Long Term status for unowned features for "scheduled for deletion" where applicable

* Relax wording about scheduled for deletion features

* Delete blob granule feature.  WIP.  Does not compile.

* more incremental hacking to remove / comment out blob granule related code

* more hacking to remove blob granule related code, e.g. blob manager and blob migrator roles

* delete more blob granule stuff

* more hacking

* more hacking

* more hacking

* More changes to remove blob granule related code.  IT COMPILES NOW

* dont try to run AuthzSecurity tests as we have deleted that workload as part of this effort

* delete more stuff that matches, abbreviates, or smells like blob granule related

* EncryptKeyProxy: dont do blobMetadata stuff, because that is not used and support is being removed

* delete more references to blob granule stuff

* SimulationConfig::setEncryptionAtRestMode: always use DISABLED; also disable EncryptKeyProxyTest.toml

* format code

* manual update to bindings/java/src/tests.cmake to remove a deleted file

* fix compile errors.  I guess by default I dont build Java bindings

* remove unneeded blob granule functions rather than #if..#endif them out

* remove more code in #if..#endif

* remove more code in #if 0..#endif

* revert changes to fdb_c.h in preparation for marking removed API calls as removed

* rework C API declarations to in preparation for marking blob granule APIs as removed

* deprecate removed glob granule related API functions as of version 740 (and add a comment to request a justification of this convention)

* make progress on broken ctests.  E.g. 1) python does not need to do blob granule stuff.  2) authz tests seemingly not needed

* remove blob granule stuff from Java and Python APIs and fix test runner stuff so that ctests pass

* reformat comments to fix compile error.  FIXME: why is this error not happening on the default compile commands we use

* hacks all the way down to try to fix the Mac build

* add pointed comment about the perceived pointlessness of the API deprecation scheme embodied in this source file

* really serious about the C++ style comments, arent we

* remove commented-out code from prior iterative efforts

* put back undeleted code in original order

* delete commented-out code

* update feature-status.md to say blob granule is mostly deleted

* upgrade `mostly deleted` to `has been deleted`
2025-10-13 16:18:56 -07:00