f51fca0bf3
FastRestore:Sanity check actors do not throw error silently
2020-06-05 17:44:24 -07:00
0a49dbd6d0
FastRestore:Check restore request must have been set when master needs it
2020-05-12 19:27:46 -07:00
f5aef706f6
FastRestore:Delay leader election until restore requests are set
2020-05-12 19:11:08 -07:00
a833724322
Merge pull request #3078 from xumengpanda/mengxu/fr-circus-stall-PR
...
Performant restore: Various improvements based on circus test
2020-05-07 20:07:23 -07:00
ff992060cd
Merge pull request #3073 from tclinken/fix-open-for-ide-build
...
Fix non-boost-related OPEN_FOR_IDE build errors
2020-05-07 14:47:59 -07:00
be6cfa4b5a
Add batchIndex in loading files
2020-05-05 19:29:53 -07:00
e4bf6d570f
FastRestore:Add assertion and trace events for diagnosis
2020-05-05 19:12:15 -07:00
9000a4a548
FastRestoreMaster:Free samples memory immediately after use
2020-05-04 11:49:36 -07:00
0ba1551116
FastRestore:Trace memory usage periodically
2020-05-04 11:20:53 -07:00
943e8e7e84
More fixes for OPEN_FOR_IDE build
2020-05-01 23:04:12 -07:00
f9f1ac6594
FastRestore:Revise TraceEvent for better diagnosis
2020-05-01 16:31:55 -07:00
ba07d449f6
FastRestore:Add trace to measure vb time
2020-05-01 13:30:45 -07:00
41c0a1768f
FastRestore:Make FastRestore event type more descriptive
2020-05-01 10:27:08 -07:00
07a9a05683
FastRestore:Agent:Fix restore requests
2020-04-30 16:20:20 -07:00
b315163033
Fix a memory corruption error
2020-04-28 15:39:23 -07:00
9bfc5bbea8
Check RestorableFileSet's key ranges in simulation
...
Ranges written in the manifest file should match with actual file content.
2020-04-21 12:55:40 -07:00
7e890cb6be
FastRestore:Minor simplify code
2020-04-17 15:00:07 -07:00
52dba96f07
Fix:Segmentation fault when get status in BackupAndParallelRestore workload
2020-04-17 13:40:32 -07:00
2d9e9a0502
FastRestore:Use knob to guard the expensive way to get range versions
2020-04-17 10:02:58 -07:00
0342a046bb
FastRestore:Master:Use minRangeVersion to initialize rangeVersions
2020-04-17 09:32:19 -07:00
d6c1baa784
FastRestore:Filter out log mutations whose version is smaller than range mutation version
2020-04-15 19:45:03 -07:00
4d06e837dc
Remove getPartitionedRestoreSet() API
...
Use getRestoreSet() instead for both old and new partitioned logs.
2020-04-08 20:12:09 -07:00
0cf6013357
Refactor to remove describePartitionedBackup()
...
The backup container can figure out if partitioned logs are used by looking at
mutation logs, thus consolidating the API to a single describeBackup() as
before.
2020-04-08 15:59:37 -07:00
5ebafdb94c
FastRestore:Apply clang-format to changes
2020-04-07 15:57:03 -07:00
e5b2cd81d5
FastRestore:Cleanup debug code
2020-04-07 15:56:44 -07:00
536e65cd76
FastRestore:Introduce debugFRMutation for debug keys
2020-04-05 15:00:36 -07:00
a81ec332a9
FastRestore:Fix:Master cannot throttle on in progress version batches when it release batches out of order in simulation
2020-04-04 17:34:26 -07:00
13f343ec96
Resolve minor review comment
2020-03-28 16:03:01 -07:00
8a30526336
FastRestore:Remove commented assertion
2020-03-28 13:11:32 -07:00
404a3e2619
FastRestore:Loader:Remove sanity chech for the order of sending log and range mutations
2020-03-27 23:36:13 -07:00
21a5c67f9a
FastRestore:Remove assertion on mutation sending order
2020-03-27 17:09:31 -07:00
75fc9af5c8
Apply clang format
2020-03-27 16:55:52 -07:00
0222e8096c
FastRestore:Send log mutations and range mutations in parallel
...
With the subversion extension, appliers can order log and range mutations
based on LogMessageVersion instead of sending order.
2020-03-27 16:54:19 -07:00
97f8e46388
Sanity check subversion for log mutations
2020-03-27 13:07:08 -07:00
32b0ba1822
Merge branch 'master' into mengxu/parallel-range-log-file-loading-PR
2020-03-27 12:13:47 -07:00
6299ad3913
FastRestore:Load range and log files in parallel for new backup format
2020-03-26 13:17:44 -07:00
772ab70aee
Add an option for fast restore to restore old backups
...
If "usePartitionedLogs" is set to false, then the workload uses old backups for
restore.
2020-03-26 13:04:00 -07:00
feedab02a0
Merge pull request #2855 from xumengpanda/mengxu/fr-api-atomicrestore-PR
...
Add ApiCorrectnessAtomicRestore workload for the new performant restore
2020-03-25 18:05:26 -07:00
120272f025
Change unlockDB from RestoreMaster to Agent
2020-03-25 11:04:49 -07:00
ca8966a28b
Move lockDB into submitRestore request from restore worker
...
AtomicRestore needs to lock DB before we start the restore worker.
So we cannot lock DB in restore worker with a different randomUID.
2020-03-24 23:39:35 -07:00
97702d91c8
Skip recruiting backup workers for older epochs before min backup version
...
When master starts recruiting backup workers, if there is no active backup job
or the min version of the backup job is greater than old epoch's end version,
then these old epochs can be skipped.
2020-03-21 13:44:02 -07:00
6b0d2923e7
Add target version as the limit for version batches
...
If using partitioned logs, the mutations after the target version can be
included if this limit is not considered.
2020-03-20 20:15:09 -07:00
799f0b4b0e
Small code refactor
2020-03-20 20:15:09 -07:00
1a1f572f29
Fix a time gap for monitoring backup keys
...
Backup worker starts by check if there are backup keys and then runs
monitorBackupKeyOrPullData() loop, which does the check again. The second check
can be delayed, which causes the loop to perform NOOP pops. The fix removes
this second check and uses the result of the first check to decide what to do
in the loop.
2020-03-20 20:15:09 -07:00
6fb7316185
Fix asset end version if request.targetVersion is -1
2020-03-20 20:15:09 -07:00
d82432da3c
Fix wrong end version for restore loader
...
The restore cannot exceed the target version of the restore request. Otherwise,
the version restored is larger than the requested version.
2020-03-20 20:15:08 -07:00
1f95cba53e
Add describePartitionedBackup() for parallel restore
...
For partitioned logs, computing continuous log end version from min logs begin
version. Old backup test keeps using describeBackup() to be correctness clean.
Rename partitioned log file so that the last number is block size.
2020-03-20 20:13:38 -07:00
940bea102a
Add a knob to switch mutation logs for parallel restore
...
Knob FASTRESTORE_USE_PARTITIONED_LOGS, default is true to enable partitioned
mutation logs. Otherwise, old mutation logs are used.
2020-03-20 20:13:38 -07:00
88ad28e576
Integrate parallel restore with partitioned logs
...
In parallel restore, use new getPartitionedRestoreSet() to get a set containing
partitioned mutation logs. The loader uses a new parser to extract mutations
from partitioned logs.
TODO: fix unable to restore errors.
2020-03-20 20:13:38 -07:00
d3071409c5
FastRestore:Add comment for integrating with new backup format
2020-03-20 20:13:38 -07:00
Meng Xu