4bf579a6d5
FastRestore:Fix race condition in pipeline
...
Master should not start asking appliers to apply mutations at batchIndex
until all appliers have applied mutations at (batchIndex - 1).
Otherwise, mutations may not be applied in increasing order of versions,
because appliers at different batch index can have overlapped key ranges.
2020-01-23 16:34:45 -08:00
e011f39829
FastRestore:Add sanity check and trace events
2020-01-23 16:03:41 -08:00
009fcdeb16
FastRestore:Sanity check each restore asset is processed exactly once
2020-01-21 17:17:45 -08:00
022783b449
Start batches in reverse order for testings and code cleanup
2020-01-21 14:49:40 -08:00
4ac92d223b
Cleanup batch buffer for each restore request
2020-01-21 14:49:36 -08:00
1a130b0df3
FastRestore:Fix race condition on handleApplyToDBRequest
2020-01-17 17:01:09 -08:00
bfbf2164c4
FastRestore:Applier buffer data for multiple batches
2020-01-17 17:01:01 -08:00
67e913c3d5
Change LoadingParam struct and endVersion definition
...
1) Remove endVersion field because it has been included in RestoreAsset;
2) Ensure endVersion in VersionBatch and RestoreAsset is always exclusive;
3) Revise ASSERT in laoder and applier in situations when the dummy commit version
is endVersion, to avoid false positive ASSERT failure.
2020-01-07 11:48:03 -08:00
8d6f511816
FastRestore:Resolve review comment
...
Filter out range mutations that do not overlap with the restore range.
Small changes on format.
2019-12-22 20:09:10 -08:00
ddcf3fdd80
FastRestore:Apply clang format
2019-12-20 22:00:36 -08:00
d888e3100b
FastRestore:Applier:Add invariant
2019-12-20 19:34:28 -08:00
e98b2a0d1c
FastRestore:Introduce RestoreAsset
2019-12-20 18:00:10 -08:00
1371db4cdc
FastRestore:Self code review and cleanup
...
1. Review memory use cases and improve:
Ensure state varialble is initialized and
change unnecessary state variable to variable.
2. Remove debug code that is no longer useful;
3. Mute verbose debug.
2019-12-11 16:37:33 -08:00
feb2a8c70c
FastRestore Change RestoreSendMutationVectorVersionedRequest name
...
Change RestoreSendMutationVectorVersionedRequest to
RestoreSendVersionedMutationsRequest for better naming
2019-12-10 17:23:40 -08:00
39a4f2372f
Change FASTRESTORE_SAMPLING_PERCENT to 0 to 100
2019-12-04 21:26:27 -08:00
c6b36dbffb
FastRestore:Sampling:Resolve review comments
2019-12-04 17:35:11 -08:00
2b987d1945
FastRestore:typedef Standalone<VectorRef<MutationRef>> MutationsVec
2019-12-04 11:39:55 -08:00
9383c3f0a6
FastRestore:Sampling:Apply clang format
2019-12-03 21:27:06 -08:00
3310f67e9e
Merge branch 'mengxu/fast-restore-fix-valgrind-PR' into mengxu/fast-restore-sampling-PR
2019-12-03 16:24:40 -08:00
530b689299
Move state variable to the start of function
2019-11-26 11:17:59 -08:00
474f0067c4
Remove unneeded state
2019-11-25 23:10:14 -08:00
bb97307f08
FastRestore:Applier:Move state variables at the start of actor
2019-11-25 21:25:14 -08:00
ae7e42face
Merge pull request #2313 from xumengpanda/mengxu/fastrestore-applyToDB-bugfix-PR
...
Performant restore [8/XX]: Fix bugs in applyToDB logic and add more tests
2019-11-12 08:50:23 -08:00
630c29d160
FastRestore:resolve review comments
...
1) wait on whenAtLeast;
2) Put BigEndian64 into the function call and the decoder to prevent
future people from making the same mistake.
2019-11-11 17:00:16 -08:00
eb67886b75
FastRestore:Move comment to func definition
...
Resolve review comments.
2019-11-11 15:10:27 -08:00
e345c9061f
FastRestore:Refine debug messages
2019-11-04 11:47:38 -08:00
27c7ef09a3
FastRestore:Revise code in self review
...
When we read the txnId from decodeRestoreApplierKey func,
we should convert the integer to little endian.
2019-11-03 20:43:17 -08:00
58aa6711e4
FastRestore:ApplyToDB:BugFix:Serialize integer as bigEndian to ensure lexico order
2019-11-03 17:26:07 -08:00
c53f817c5e
FastRestore:Convert handleInitVersionBatchRequest to plain func
2019-10-24 13:06:50 -07:00
b1881a7c1c
FastRestore:Apply clang-format
2019-10-23 20:49:14 -07:00
bae0c907a6
FastRestore:Convert unnecessary actor function to plain function
2019-10-23 15:10:34 -07:00
ba7e499efe
FastRestore:AtomicOpTest:Limit 1 actor per client
2019-10-23 14:04:14 -07:00
41f0cd624b
FastRestore:Applier:Use shouldCommit to replace the duplicate code
2019-10-23 13:36:19 -07:00
96d463bab6
FastRestore:Fix bug in applying mutations and increase atomicOp test worload
...
When Applier applies mutations to the destination cluster, it advances the
mutation cursor twice when it should only advance it once.
This makes restore miss some mutations when the applying txn includes
more than 1 mutations.
2019-10-22 23:24:23 -07:00
2dbbce55a8
FastRestore:Applier:Mute debug trace
2019-10-21 14:36:07 -07:00
4af69fd94f
Merge branch 'master' into mengxu/fastrestore-multifiles-has-sameversion-mutations-PR-testPR
2019-10-21 14:35:04 -07:00
f08ad48b7b
FastRestore:Applier:handleSendMutationVectorRequest:Add comment
2019-10-21 14:31:21 -07:00
4efddc9b89
FastRestore:Applier:Reduce LoC
...
When a key does not exist in a map, it is created by default when it is accessed by []
2019-10-21 14:31:21 -07:00
6f1ecd1b11
FastRestore:handleSendMutationVectorRequest:Receive mutations in order of versions
2019-10-21 14:31:21 -07:00
ab946eb24f
FastRestore:Applier:Turn on debug
2019-10-19 17:07:31 -07:00
0fe0a14987
FastRestore:handleSendMutationVectorRequest:Receive mutations in order of versions
2019-10-17 17:21:00 -07:00
af8047e79b
FastRestore:ApplyToDB:Change state variable to variable
2019-10-14 16:38:01 -07:00
0c8de91932
FastRestore:applyToDB:Add functions to DBApplyProgress for encapsulation
2019-10-14 16:24:36 -07:00
f89b5586df
FastRestore:applyToDB:Record applyToDB progress in DBApplyProgress struct
...
This avoids repetitive code
2019-10-14 14:57:17 -07:00
7b36fee38f
FastRestore:applyToDB:Cosmic change for review comments
...
No functional change.
2019-10-14 12:52:32 -07:00
71509a5157
FastRestore:Applier:applyToDB:Clang format
2019-10-10 17:36:38 -07:00
48e0620e5f
FastRestore:Applier:applyToDB:Handle txn with errors
2019-10-10 17:24:07 -07:00
d160810662
FastRestore:Resolve review comments
2019-09-04 16:48:43 -07:00
3b54363780
FastRestore:Apply Clang-format
2019-08-01 18:09:12 -07:00
b0c31f28af
FastRestore:Fix bug that blocks restore
...
1) Should recruit only configured number of roles;
2) Should never register a restore master interface as a restore worker (loader or applier) interface.
2019-07-25 17:55:37 -07:00
Meng Xu