4bf579a6d5
FastRestore:Fix race condition in pipeline
...
Master should not start asking appliers to apply mutations at batchIndex
until all appliers have applied mutations at (batchIndex - 1).
Otherwise, mutations may not be applied in increasing order of versions,
because appliers at different batch index can have overlapped key ranges.
2020-01-23 16:34:45 -08:00
e011f39829
FastRestore:Add sanity check and trace events
2020-01-23 16:03:41 -08:00
bffb4e380d
AtomicOp:Debug
2020-01-22 15:04:25 -08:00
c4a11597cf
FastRestore:Revise sanity check condition
2020-01-22 13:27:25 -08:00
52e3d20d39
FastRestore:VersionBatch replace vector with set
...
In order to ensure each backup file only appears in version batch once.
2020-01-22 13:13:10 -08:00
5893f66ced
FastRestore:Deduplicate backup range and log files
2020-01-21 21:55:02 -08:00
4fbbff8ccd
FastRestore:Cannot rely on default enum value
...
enum class value does not have a standard default value.
We cannot assume the default value is 0 for scala enum type.
2020-01-21 21:43:32 -08:00
009fcdeb16
FastRestore:Sanity check each restore asset is processed exactly once
2020-01-21 17:17:45 -08:00
022783b449
Start batches in reverse order for testings and code cleanup
2020-01-21 14:49:40 -08:00
4ac92d223b
Cleanup batch buffer for each restore request
2020-01-21 14:49:36 -08:00
e933716109
FastRestore:Enable multiple batch pipelining
2020-01-17 17:01:09 -08:00
8d3f3aa926
FastRestore:Pipeline multiple version batches
2020-01-17 17:01:06 -08:00
441f3e2814
FastRestore:Master buffer data and progress for each batch
2020-01-17 17:01:06 -08:00
d69bd2f661
FastRestore:Loader buffer data for multiple batches
2020-01-17 17:01:06 -08:00
bfbf2164c4
FastRestore:Applier buffer data for multiple batches
2020-01-17 17:01:01 -08:00
f436ea806e
FastRestore:Resolve review comment
...
1) Sort logfiles by endVersion
2) Exit program early when restore will not succeed
3) Do not increase nextVersion unncessarily when
calculate version batches.
4) Change assert condition that ensures progress in
calculating version batches.
2020-01-13 14:08:27 -08:00
dba85d28fc
FastRestore:Cosmetic revision
2020-01-08 10:53:53 -08:00
a2b26906e8
FastRestore:Filter out empty files before distributing workload
...
and clean up unused code
2020-01-07 17:01:53 -08:00
c29e380076
FastRestore:Remove prevVersion from LoadingParam
2020-01-07 14:59:17 -08:00
9df02512ab
FastRestore:Apply clang-format
2020-01-07 11:50:32 -08:00
67e913c3d5
Change LoadingParam struct and endVersion definition
...
1) Remove endVersion field because it has been included in RestoreAsset;
2) Ensure endVersion in VersionBatch and RestoreAsset is always exclusive;
3) Revise ASSERT in laoder and applier in situations when the dummy commit version
is endVersion, to avoid false positive ASSERT failure.
2020-01-07 11:48:03 -08:00
c3f8f3b445
FastRestore:Build VersionBatch less than threshold size
2020-01-07 11:46:56 -08:00
61b29de3ce
FastRestore:Self code review
...
Clean up commented code;
Add sanity check.
2019-12-20 22:24:34 -08:00
e98b2a0d1c
FastRestore:Introduce RestoreAsset
2019-12-20 18:00:10 -08:00
650be617f1
FastRestore:Add tests to CMakefile
2019-12-12 10:32:13 -08:00
b5d7890ce0
FastRestore:Resolve review comments
2019-12-12 07:45:30 -08:00
9670d64fbd
FastRestore:Remove commented code
2019-12-11 16:48:40 -08:00
1371db4cdc
FastRestore:Self code review and cleanup
...
1. Review memory use cases and improve:
Ensure state varialble is initialized and
change unnecessary state variable to variable.
2. Remove debug code that is no longer useful;
3. Mute verbose debug.
2019-12-11 16:37:33 -08:00
20a19978f9
FastRestore:LoadingParam cleanup
2019-12-10 17:20:44 -08:00
9383c3f0a6
FastRestore:Sampling:Apply clang format
2019-12-03 21:27:06 -08:00
6b07c271f1
Fix non-deterministic error
2019-12-03 16:55:23 -08:00
3310f67e9e
Merge branch 'mengxu/fast-restore-fix-valgrind-PR' into mengxu/fast-restore-sampling-PR
2019-12-03 16:24:40 -08:00
153b713b53
FastRestore:Add sampling on parsed mutations
2019-12-03 12:52:17 -08:00
78f10f15b3
FastRestore:replace insert with emplace for map and vector
...
This resolves the review suggestions.
2019-11-21 22:47:04 -08:00
3f5491318d
FastRestore:Fix bug that cause nondeterminism
...
1) Use map iterator instead of pointer to maintain stability when map is inserted or deleted
2) dummySampleWorkload: clear rangeToApplier data in each sampling phase. otherwise, we can
have an increasing number of keys assigned to the applier.
2019-11-15 11:30:09 -08:00
9e36b897e6
FastRestore:Loaders must send to appliers log files data before range files
2019-11-12 21:43:12 -08:00
592f4c0fc4
FastRestore:Remove RestoreSetApplierKeyRangeVectorRequest
2019-11-12 17:59:11 -08:00
7e4c4ea98e
FastRestore:Load mutations before assign ranges to appliers
2019-11-12 17:14:17 -08:00
e345c9061f
FastRestore:Refine debug messages
2019-11-04 11:47:38 -08:00
d4de608bb6
Fix OPEN_FOR_IDE build
2019-10-25 10:42:22 -07:00
6f1ecd1b11
FastRestore:handleSendMutationVectorRequest:Receive mutations in order of versions
2019-10-21 14:31:21 -07:00
17821a1424
FastRestore:unlockDatabase should always succeed
2019-10-17 00:50:13 -07:00
3e2b3de4d0
FastRestore:RestoreMaster:Remove the extra lockDatabase in RestoreMaster
2019-10-17 00:50:13 -07:00
2cd7010efb
FastRestore:Add fileIndex to RestoreFileFR struct and bug fix
...
Fix bugs in RestoreMaster that cannot properly lock or unlock DB when
exception occurs;
Fix bug in ordering backup files
2019-10-17 00:50:13 -07:00
048c341a7d
FastRestore:Bug fix after merge with master
...
Include RestoreWorkerInterface.h instead of RestoreInterface.h into fdbserver.actor.cpp
Report warning instead of error when unlockDatabase throws error.
2019-09-04 21:14:09 -07:00
d160810662
FastRestore:Resolve review comments
2019-09-04 16:48:43 -07:00
2602cb3591
FastRestore:Rename RestoreConfig to RestoreConfigFR to fix link problem in windows
...
Because the current restore has defined RestoreConfig, windows linker complains.
This commit rename the RestoreConfig used in FastRestore as RestoreConfigFR.
2019-08-02 23:00:12 -07:00
3b54363780
FastRestore:Apply Clang-format
2019-08-01 18:09:12 -07:00
93680cddd6
FastRestore:Remove redundant constructFilesWithVersionRange
2019-07-29 16:32:48 -07:00
b0c31f28af
FastRestore:Fix bug that blocks restore
...
1) Should recruit only configured number of roles;
2) Should never register a restore master interface as a restore worker (loader or applier) interface.
2019-07-25 17:55:37 -07:00
Meng Xu