Commit Graph

25 Commits

Author SHA1 Message Date
Jingyu Zhou 5e8feac8c9 Fix ConsistencyCheck_InvalidShardSize due to unfair split
ConsistencyCheck thinks the shard is too large and should be split. However, it
is possible that the split point returned by the Storage Server is discarded
due to "unfair" sizes after the split. Fix this problem by checking if this is
the case in ConsistencyCheck.
2023-04-28 11:35:05 -07:00
Steve Atherton 858b51a69b Address review comments. KeyRangeMapSnapshot is now ReferenceCounted and getSnapshot() returns a Reference to discourage copying. Added several comments for clarity. Added FormatUsingTraceable and changed all new formatters to use it except for Standalone<T> which redirects to the formatter for T. 2023-04-24 19:01:05 -07:00
Steve Atherton c57ed25987 Renamed SystemDBLockWriteNow() to SystemDBWriteLockedNow() and changed definition to be more direct / clear. 2023-04-22 13:17:41 -07:00
Steve Atherton 639d4d05ef Removed SYSTEM_PRIORITY_IMMEDIATE from KeyBackedTypes and all options from KeyBackedRangeMap database functions. Added SystemTransactionGenerator<> for wrapping Database types and generating transactions with selected system level options. 2023-04-21 19:00:29 -07:00
Steve Atherton ebb3c8d698 Apply clang-format, debug output changes. 2023-04-18 22:21:27 -07:00
Steve Atherton bc6b9cb83f Added toJSON for DD Range configuration. 2023-04-18 22:21:27 -07:00
Steve Atherton 53ee26d758 Changed KeyBackedTypes to an actor file. Added TypedKeySelectors for Map and Set classes and getRange() keySelector methods. Added debug macro for KeyBackedTypes. Rewrote KeyBackedRangeMap using keyselectors on KeyBackedMap. 2023-04-18 22:21:19 -07:00
Steve Atherton b7e68bbf51 DDConfiguration class for modeling user specified key range configuration options. Added KeyBackedRangeMapSnapshot, some other supporting changes to KeyBackedTypes. Added invalidKey to give KeyBackedTypes a safe prefix to avoid accidental userspace modification from uninitialized accessors. 2023-04-18 22:09:18 -07:00
Dan Adkins 02f0a44987
Avoid divide-by-zero in isKeyValueInSample. (#9799)
In the pathological case that both key size and value size are 0,
the probability of choosing that key-value pair is 0, and we divide
by zero when computing the sampledSize.

This change adds documentation to that function, which was quite
difficult to understand. In addition, we add `probability` to the
returned values, since one of the callers was backing it out from
sampledSize and itself in danger of dividing by zero.
2023-03-24 16:05:26 -04:00
Evan Tschannen a258d775c3
Merge pull request #9710 from sfc-gh-etschannen/feature-custom-dd
Added the ability to manually create a shard and also increase its replication factor
2023-03-20 15:35:10 -07:00
A.J. Beamon dc2bd78aa7 The consistency check should retry if it couldn't find all the commit proxies when getting key server locations 2023-03-17 12:00:47 -07:00
Evan Tschannen de7f40d2f4 made a variety of improvements that came from code review 2023-03-17 10:30:34 -07:00
Evan Tschannen ac54962533 code cleanup 2023-03-16 09:47:21 -07:00
Evan Tschannen aaf7b9b32b Added the ability to manually create a shard and also increase its replication factor 2023-03-15 11:26:15 -07:00
Evan Tschannen 4a17ed363a Fix: the consistency check did not properly report failed tests 2023-03-08 16:56:23 -08:00
Jingyu Zhou 496b29aa3c
Update fdbserver/ConsistencyScan.actor.cpp 2023-01-23 09:34:04 -08:00
Jingyu Zhou 78838f1c74 Fix a consistency scan infinite looping
If a storage server was removed, the consistency scan may be looping forever,
because "request_maybe_delivered" was not handled.
2023-01-14 09:33:00 -08:00
A.J. Beamon 3e9b6ce937 Don't disallow a seed value of 0 2022-12-02 15:32:34 -08:00
A.J. Beamon 304747d387 The simulator was changing the configuration of the cluster while the consistency check was expecting a quiescent database 2022-11-03 08:35:34 -07:00
Jingyu Zhou c127bb1c30 Fix some clang warnings on unused variables 2022-11-01 15:38:47 -07:00
Xiaoxi Wang bb0236433c
Merge pull request #8540 from sfc-gh-xwang/feature/main/storageMetrics
Make MockStorageServer serve StorageMetrics related request
2022-10-25 17:29:21 -07:00
Xiaoxi Wang 3c67b7df39 extract serveStorageMetricsRequests template function 2022-10-24 09:58:41 -07:00
Vishesh Yadav aa99b89d53 Don't fail ConsistencyCheck on first mismatch
ConsistencyCheck fails when it sees the first corrupted shard. We may want to
keep it running so that we can see all the corrupted data in logs.
2022-10-21 17:02:02 -07:00
Sreenath Bodagala e83229ed8b
- Version vector specific: Propagate the latest commit version, as (#8393)
part of the read request, when reading from a storage server.
2022-10-05 16:23:56 -04:00
sfc-gh-ngoyal 1bd97fe628
Recruit new singleton for consistency checker. (#5804)
* Recruit new singleton for consistency checker.

* Recruit the consistency checker only if enabled.

* Add a yield in monitorConsistencyChecker().

* Minor fixes.

* Consistency check workload enhancements.

* Minor fixes and clarifications.

* clang format

* Clang format.

* Minor fixes, cleanup, debug tracing.

* Misc.

* Move the consistency scan information from dbconfig to a key backed object.

* Move consistency scan config out of db cofig to a state object and feature rename.

* ConsistencyCheck workload refactor.

* devFormat

* Update fdbcli/ConsistencyScanCommand.actor.cpp

* Review Comments.

Co-authored-by: negoyal <neelam.goyal@gmail.com>
Co-authored-by: Ata E Husain Bohra <ata.husain@snowflake.com>
2022-09-16 09:03:06 -07:00