wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,885 Commits 36 Branches 335 Tags 299 MiB
Commit Graph

270 Commits

Author SHA1 Message Date
Jingyu Zhou df064ac922
Merge pull request #3321 from xumengpanda/mengxu/fr-restore-ranges-PR 
Fast Restore: Support restoring sub ranges in the framework
 2020-06-09 21:01:39 -07:00
Meng Xu 8533daac50 parallelRestoreFinish:fix used_during_commit error 2020-06-09 15:40:24 -07:00
Meng Xu e5e4cb5c90 parallelRestoreFinish:separate check restore finished and clear restoreRequestDoneKey 2020-06-09 11:32:20 -07:00
Meng Xu d85dc5a4d3 FastRestore:Only clear ranges that will be restored 
Instead of clearning the entire normal key space.

This commit also removes some unnecessary tr->reset() which can invalid the txn backoff time.
 2020-06-08 22:41:49 -07:00
Meng Xu e7fdd614fe Remove assert on error_code_key_not_found in previous commit 
Correctnss shows the assert does not hold in all scenarios
 2020-06-08 19:02:56 -07:00
Meng Xu 5022566b35 Validate if key_not_found error ever happens 2020-06-08 16:59:00 -07:00
Jingyu Zhou a833724322
Merge pull request #3078 from xumengpanda/mengxu/fr-circus-stall-PR 
Performant restore: Various improvements based on circus test
 2020-05-07 20:07:23 -07:00
Meng Xu 7b5d43da9c FastRestore:Remove unused field in RestoreRequest 2020-05-03 20:59:47 -07:00
tclinken 494d1d3518 Fixed unused variable warning in atomicParallelRestore 2020-05-01 18:14:51 -07:00
Meng Xu a998fa1e1a
Merge pull request #3053 from xumengpanda/mengxu/fr-bc-fix-PR 
Fix performant restore bug introduced in PR 3041
 2020-04-29 17:09:26 -07:00
Meng Xu 3f510d0653
Merge pull request #3036 from jzhou77/backup-cmd 
Several bug fixes for new backups
 2020-04-29 16:34:59 -07:00
Jingyu Zhou 9da76c35ea Fix a memory corruption error 
The backup container URL should be Key instead of KeyRef.
 2020-04-29 15:55:34 -07:00
Meng Xu 33dedaab35 FastRestore:Fix submitParallelRestore when targetVersion is unset 2020-04-29 15:44:54 -07:00
Jingyu Zhou 364142d02c Fix backupWorkerEnabled flag not set bug for first backup 
This can cause restore failures because the latestLogEndVersion could be wrong.
 2020-04-28 15:39:24 -07:00
Jingyu Zhou 7d59e53349 Consolidate makePadding() 2020-04-28 15:39:23 -07:00
Meng Xu f5e8345496 FastRestoreAgent:Use atomicParallelRestore to kick off restore 
Replace the handcrafted version with atomicParallelRestore actor
which is simulation tested
 2020-04-27 22:15:00 -07:00
Jingyu Zhou 3063611355 Write range files' begin & end keys to manifest file 
This information can be very useful in knowing the content in these files,
especially for restores.
 2020-04-20 22:26:42 -07:00
Jingyu Zhou 60407bdee3 Use LiteralStringRef for backup paused key 2020-04-07 16:02:25 -07:00
Jingyu Zhou 9fb3fb9d82 Add pause/resume for new backups 
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
 2020-04-06 14:29:46 -07:00
Jingyu Zhou efbd01894d Really fix the stuck of backup 2020-03-28 17:38:02 -07:00
Jingyu Zhou f0d0c863e6 Attempt to fix 2020-03-28 17:38:02 -07:00
Jingyu Zhou 78f74ed6fe Skip copying inside BackupLogsDispatchTask 2020-03-28 17:38:02 -07:00
Jingyu Zhou a2b8be5dc6 Switch off old mutation logging on proxies for new backups 
Specifically, stop writing LogRange mutation for new backup jobs.
 2020-03-28 17:38:02 -07:00
Jingyu Zhou 9a9af7d8a8 Add more trace event details on partitioned log 2020-03-26 13:57:31 -07:00
Jingyu Zhou feedab02a0
Merge pull request #2855 from xumengpanda/mengxu/fr-api-atomicrestore-PR 
Add ApiCorrectnessAtomicRestore workload for the new performant restore
 2020-03-25 18:05:26 -07:00
Meng Xu 495afe2e0b Improve how to wati for restore to finish 
Remove default parameter for atomicRestore as suggested in review.
 2020-03-25 13:54:21 -07:00
Meng Xu 1ba11dc74b Apply clang format 2020-03-25 11:20:17 -07:00
Meng Xu 120272f025 Change unlockDB from RestoreMaster to Agent 2020-03-25 11:04:49 -07:00
Meng Xu 130b91c820 Fix segmentation fault in submitParallelRestore 2020-03-24 23:52:34 -07:00
Meng Xu ca8966a28b Move lockDB into submitRestore request from restore worker 
AtomicRestore needs to lock DB before we start the restore worker.
So we cannot lock DB in restore worker with a different randomUID.
 2020-03-24 23:39:35 -07:00
Meng Xu e59becdec8 Refactor atomicRestore for atomicParallelRestore 2020-03-24 22:37:12 -07:00
Meng Xu 241c2703c8 Fix atomicParallelRestore interface 2020-03-24 17:00:55 -07:00
Meng Xu 80d62f3cb8 Fix:Add atomicParallelRestore to header 2020-03-24 16:28:08 -07:00
Meng Xu ffb0a439ea atomicParallelRestore should not return a version 2020-03-24 16:05:09 -07:00
Meng Xu b173929316 Add atomicParallelRestore to AtomicRestore workload 2020-03-24 15:58:49 -07:00
Meng Xu 01921c3953 Move FileBackupAgent parallel restore impl to FileBackupAgentImpl 
For consistency with existing code, and
state variable can only occur in ACTOR.
 2020-03-24 15:11:34 -07:00
Meng Xu 81f7181c9e Refactor submitParallelRestore function into FileBackupAgent 2020-03-24 14:44:55 -07:00
Meng Xu 5584884c12 Refactor parallelRestoreFinish function into FileBackupAgent 2020-03-24 14:15:15 -07:00
Meng Xu 4e477f5489 Copy atomicRestore to atomicParallelRestore 2020-03-24 13:59:35 -07:00
Jingyu Zhou e1737fc644 Skip setting backupStartedKey if using old mutation logs 
For old submitBackup(), where partitionedLog is false, do not set the
backupStartedKey in BackupConfig, which signals backup workers to skip these
backups.
 2020-03-20 20:15:09 -07:00
Jingyu Zhou 38def426f4 Add a flag to submitBackup for partitioned log 
This is to distinguish with old workloads so that they can work in simulation.
 2020-03-20 20:15:08 -07:00
Jingyu Zhou e9287407d6 Backup worker updates latest log versions in BackupConfig 
If backup worker is enabled, the current epoch's worker of tag (-2,0) will be
responsible for monitoring the backup progress of all workers and update the
BackupConfig with the latest saved log version, which is the minimum version
of all tags.

This change has been incorporated in the getLatestRestorableVersion() so that
it is transparent to clients.
 2020-03-20 20:15:08 -07:00
Jingyu Zhou e15015ee6c Add mutation log version names 
I.e., BACKUP_AGENT_MLOG_VERSION for 2001 and PARTITIONED_MLOG_VERSION for 4110.
 2020-03-20 20:13:38 -07:00
Evan Tschannen 924d335aa7 Merge branch 'release-6.2' 
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	flow/Knobs.cpp
#	flow/Knobs.h
 2020-02-25 18:25:19 -08:00
Evan Tschannen d3bca19960 backup should also submit on the first proxy for similar reasons to DR 2020-02-25 15:57:32 -08:00
Markus Pilman e71fe44ee3
Merge branch 'master' into features/icc 2020-02-08 21:33:02 -08:00
Jingyu Zhou d5849af5c0 Address review comments 2020-02-05 10:33:51 -08:00
mpilman d09e07f1f5 Merge remote-tracking branch 'upstream/master' into features/icc 2020-02-04 10:26:18 -08:00
Jingyu Zhou 52c6737411 Rename backupLoggingEnabled as backupWorkerEnabled 
To highlight the changes for 7.0 backup changes. By default,
backup_worker_enabled flag is set for 7.0 version.
 2020-02-04 10:09:16 -08:00
Jingyu Zhou 0db03f1d3c Use backup_logging_enabled flag 
The default is to enable new backup workers. Users can disable this flag to
turn off the backup worker feature.
 2020-02-03 20:03:22 -08:00