wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,885 Commits 36 Branches 335 Tags 299 MiB
Commit Graph

70 Commits

Author SHA1 Message Date
Meng Xu 8533daac50 parallelRestoreFinish:fix used_during_commit error 2020-06-09 15:40:24 -07:00
Meng Xu e5e4cb5c90 parallelRestoreFinish:separate check restore finished and clear restoreRequestDoneKey 2020-06-09 11:32:20 -07:00
Jingyu Zhou 4315af25e7 Fix unfinished restore tests 
Because backup worker is disabled before performing the restore, we can't allow
a new backup of the new type running.
 2020-05-19 15:18:45 -07:00
tclinken 943e8e7e84 More fixes for OPEN_FOR_IDE build 2020-05-01 23:04:12 -07:00
Jingyu Zhou 0e54f1ed31 Fix a test failure 
minRestorableVersion may be decided by snapshot version and is larger than
contiguousLogEnd version.
 2020-04-20 22:26:42 -07:00
Meng Xu 916d361587 BackupAndParallelRestoreCorrectness:Remove unnecessary checking optional variable 2020-04-17 18:32:14 -07:00
Meng Xu 14406ad940 BackupAndParallelRestoreCorrectness:Assert on backup validity 2020-04-17 13:51:39 -07:00
Meng Xu 52dba96f07 Fix:Segmentation fault when get status in BackupAndParallelRestore workload 2020-04-17 13:40:32 -07:00
Jingyu Zhou 4e128328f7 Stop backup workers before clearing DB in parallel restore workload 
This is because the clearing of DB can be picked up by backup workers and be
applied during restore, causing restore failures.
 2020-04-11 10:26:08 -07:00
Jingyu Zhou db2cef844b Write mutation log type as a backup property 
This can solve the problem when listing log files returns empty results.
 2020-04-09 09:29:24 -07:00
Jingyu Zhou 0cf6013357 Refactor to remove describePartitionedBackup() 
The backup container can figure out if partitioned logs are used by looking at
mutation logs, thus consolidating the API to a single describeBackup() as
before.
 2020-04-08 15:59:37 -07:00
Jingyu Zhou 9fb3fb9d82 Add pause/resume for new backups 
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
 2020-04-06 14:29:46 -07:00
Meng Xu 75fc9af5c8 Apply clang format 2020-03-27 16:55:52 -07:00
Meng Xu 32b0ba1822 Merge branch 'master' into mengxu/parallel-range-log-file-loading-PR 2020-03-27 12:13:47 -07:00
Meng Xu 113d0fb48b Remove incorrect assertion 2020-03-27 12:13:30 -07:00
Jingyu Zhou 6be913a430 Add partitioned logs option to AtomicRestore workload 2020-03-26 13:04:00 -07:00
Jingyu Zhou aca458cd96 Set 50% chance to restore old backup files for fast restore 2020-03-26 13:04:00 -07:00
Jingyu Zhou 772ab70aee Add an option for fast restore to restore old backups 
If "usePartitionedLogs" is set to false, then the workload uses old backups for
restore.
 2020-03-26 13:04:00 -07:00
Jingyu Zhou feedab02a0
Merge pull request #2855 from xumengpanda/mengxu/fr-api-atomicrestore-PR 
Add ApiCorrectnessAtomicRestore workload for the new performant restore
 2020-03-25 18:05:26 -07:00
Meng Xu 1ba11dc74b Apply clang format 2020-03-25 11:20:17 -07:00
Meng Xu 120272f025 Change unlockDB from RestoreMaster to Agent 2020-03-25 11:04:49 -07:00
Meng Xu ca8966a28b Move lockDB into submitRestore request from restore worker 
AtomicRestore needs to lock DB before we start the restore worker.
So we cannot lock DB in restore worker with a different randomUID.
 2020-03-24 23:39:35 -07:00
Meng Xu 81f7181c9e Refactor submitParallelRestore function into FileBackupAgent 2020-03-24 14:44:55 -07:00
Meng Xu 5584884c12 Refactor parallelRestoreFinish function into FileBackupAgent 2020-03-24 14:15:15 -07:00
Jingyu Zhou 1155304cd5 Remove a spurious assertion 
It's possible that there is a gap between backup's contiguousLogEnd and snapshot
version.
 2020-03-23 21:39:40 -07:00
Jingyu Zhou dd90845277 Fix assert failure 
Should be backup's contiguousLogEnd > maxRestorableVersion.
 2020-03-23 14:49:05 -07:00
Meng Xu 3f31ebf659 New backup:Revise event name and explain code 2020-03-23 10:55:44 -07:00
Jingyu Zhou 524b275a94 Add a flag to submitBackup for partitioned log 
This is to distinguish with old workloads so that they can work in simulation.
 2020-03-18 16:41:35 -07:00
Jingyu Zhou f697ccd1b9 Add describePartitionedBackup() for parallel restore 
For partitioned logs, computing continuous log end version from min logs begin
version. Old backup test keeps using describeBackup() to be correctness clean.

Rename partitioned log file so that the last number is block size.
 2020-03-18 16:41:35 -07:00
Meng Xu 52ba093f23 BackupAndParallelRestoreTest:Apply Steve change in BackupCorrectness on 604ad062d5 
The backup container behavior was changed at commit 604ad062d5.
The new behavior was not applied in BackupAndParallelRestoreCorrectness.actor.cpp.
This can cause test failure due to backup cannot find a valid backup container.
 2020-02-20 08:07:03 -08:00
Meng Xu 0507e9703c FastRestore:Disable abort fast restore test 2020-02-20 06:54:14 -08:00
Meng Xu 754e2cb023 FastRestore:Fix:number of workers should be calculated 
based on number of appliers and loaders.
 2020-01-27 21:14:00 -08:00
Meng Xu 9670d64fbd FastRestore:Remove commented code 2019-12-11 16:48:40 -08:00
Meng Xu 1371db4cdc FastRestore:Self code review and cleanup 
1. Review memory use cases and improve:
Ensure state varialble is initialized and
change unnecessary  state variable to variable.

2. Remove debug code that is no longer useful;

3. Mute verbose debug.
 2019-12-11 16:37:33 -08:00
Meng Xu 4a66366a05 Use MutationsVec instead of VectorRef 2019-12-06 22:00:40 -08:00
Meng Xu 39a4f2372f Change FASTRESTORE_SAMPLING_PERCENT to 0 to 100 2019-12-04 21:26:27 -08:00
Meng Xu c6b36dbffb FastRestore:Sampling:Resolve review comments 2019-12-04 17:35:11 -08:00
Meng Xu a15320cca7 Merge branch 'master' into mengxu/fast-restore-sampling-PR 2019-12-03 21:42:01 -08:00
Meng Xu 3310f67e9e Merge branch 'mengxu/fast-restore-fix-valgrind-PR' into mengxu/fast-restore-sampling-PR 2019-12-03 16:24:40 -08:00
Meng Xu f153cadab9 ComplilationWarning:Fix actor that does not contain wait statement 2019-12-02 11:38:29 -08:00
Meng Xu 530b689299 Move state variable to the start of function 2019-11-26 11:17:59 -08:00
Meng Xu 3f5491318d FastRestore:Fix bug that cause nondeterminism 
1) Use map iterator instead of pointer to maintain stability when map is inserted or deleted
2) dummySampleWorkload: clear rangeToApplier data in each sampling phase. otherwise, we can
have an increasing number of keys assigned to the applier.
 2019-11-15 11:30:09 -08:00
Andrew Noyes de8921b660 Move RestoreWorkerInterface to fdbclient 2019-10-25 10:42:22 -07:00
Andrew Noyes d4de608bb6 Fix OPEN_FOR_IDE build 2019-10-25 10:42:22 -07:00
Meng Xu c2355f721e Merge branch 'master' into mengxu/performant-restore-PR 2019-09-04 17:11:42 -07:00
Meng Xu d160810662 FastRestore:Resolve review comments 2019-09-04 16:48:43 -07:00
Meng Xu 9cc832cfd6 FastRestore:Fix Mac and Windows compilation error 2019-08-02 14:33:08 -07:00
Meng Xu 3b54363780 FastRestore:Apply Clang-format 2019-08-01 18:09:12 -07:00
Meng Xu b0c31f28af FastRestore:Fix bug that blocks restore 
1) Should recruit only configured number of roles;
2) Should never register a restore master interface as a restore worker (loader or applier) interface.
 2019-07-25 17:55:37 -07:00
Meng Xu 45083edf74 Merge branch 'master' into mengxu/performant-restore-PR 
Fix conflicts as well.
 2019-07-25 10:46:11 -07:00