a998fa1e1a
Merge pull request #3053 from xumengpanda/mengxu/fr-bc-fix-PR
...
Fix performant restore bug introduced in PR 3041
2020-04-29 17:09:26 -07:00
3f510d0653
Merge pull request #3036 from jzhou77/backup-cmd
...
Several bug fixes for new backups
2020-04-29 16:34:59 -07:00
9da76c35ea
Fix a memory corruption error
...
The backup container URL should be Key instead of KeyRef.
2020-04-29 15:55:34 -07:00
33dedaab35
FastRestore:Fix submitParallelRestore when targetVersion is unset
2020-04-29 15:44:54 -07:00
364142d02c
Fix backupWorkerEnabled flag not set bug for first backup
...
This can cause restore failures because the latestLogEndVersion could be wrong.
2020-04-28 15:39:24 -07:00
7d59e53349
Consolidate makePadding()
2020-04-28 15:39:23 -07:00
f5e8345496
FastRestoreAgent:Use atomicParallelRestore to kick off restore
...
Replace the handcrafted version with atomicParallelRestore actor
which is simulation tested
2020-04-27 22:15:00 -07:00
3063611355
Write range files' begin & end keys to manifest file
...
This information can be very useful in knowing the content in these files,
especially for restores.
2020-04-20 22:26:42 -07:00
60407bdee3
Use LiteralStringRef for backup paused key
2020-04-07 16:02:25 -07:00
9fb3fb9d82
Add pause/resume for new backups
...
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
2020-04-06 14:29:46 -07:00
efbd01894d
Really fix the stuck of backup
2020-03-28 17:38:02 -07:00
f0d0c863e6
Attempt to fix
2020-03-28 17:38:02 -07:00
78f74ed6fe
Skip copying inside BackupLogsDispatchTask
2020-03-28 17:38:02 -07:00
a2b8be5dc6
Switch off old mutation logging on proxies for new backups
...
Specifically, stop writing LogRange mutation for new backup jobs.
2020-03-28 17:38:02 -07:00
9a9af7d8a8
Add more trace event details on partitioned log
2020-03-26 13:57:31 -07:00
feedab02a0
Merge pull request #2855 from xumengpanda/mengxu/fr-api-atomicrestore-PR
...
Add ApiCorrectnessAtomicRestore workload for the new performant restore
2020-03-25 18:05:26 -07:00
495afe2e0b
Improve how to wati for restore to finish
...
Remove default parameter for atomicRestore as suggested in review.
2020-03-25 13:54:21 -07:00
1ba11dc74b
Apply clang format
2020-03-25 11:20:17 -07:00
120272f025
Change unlockDB from RestoreMaster to Agent
2020-03-25 11:04:49 -07:00
130b91c820
Fix segmentation fault in submitParallelRestore
2020-03-24 23:52:34 -07:00
ca8966a28b
Move lockDB into submitRestore request from restore worker
...
AtomicRestore needs to lock DB before we start the restore worker.
So we cannot lock DB in restore worker with a different randomUID.
2020-03-24 23:39:35 -07:00
e59becdec8
Refactor atomicRestore for atomicParallelRestore
2020-03-24 22:37:12 -07:00
241c2703c8
Fix atomicParallelRestore interface
2020-03-24 17:00:55 -07:00
80d62f3cb8
Fix:Add atomicParallelRestore to header
2020-03-24 16:28:08 -07:00
ffb0a439ea
atomicParallelRestore should not return a version
2020-03-24 16:05:09 -07:00
b173929316
Add atomicParallelRestore to AtomicRestore workload
2020-03-24 15:58:49 -07:00
01921c3953
Move FileBackupAgent parallel restore impl to FileBackupAgentImpl
...
For consistency with existing code, and
state variable can only occur in ACTOR.
2020-03-24 15:11:34 -07:00
81f7181c9e
Refactor submitParallelRestore function into FileBackupAgent
2020-03-24 14:44:55 -07:00
5584884c12
Refactor parallelRestoreFinish function into FileBackupAgent
2020-03-24 14:15:15 -07:00
4e477f5489
Copy atomicRestore to atomicParallelRestore
2020-03-24 13:59:35 -07:00
e1737fc644
Skip setting backupStartedKey if using old mutation logs
...
For old submitBackup(), where partitionedLog is false, do not set the
backupStartedKey in BackupConfig, which signals backup workers to skip these
backups.
2020-03-20 20:15:09 -07:00
38def426f4
Add a flag to submitBackup for partitioned log
...
This is to distinguish with old workloads so that they can work in simulation.
2020-03-20 20:15:08 -07:00
e9287407d6
Backup worker updates latest log versions in BackupConfig
...
If backup worker is enabled, the current epoch's worker of tag (-2,0) will be
responsible for monitoring the backup progress of all workers and update the
BackupConfig with the latest saved log version, which is the minimum version
of all tags.
This change has been incorporated in the getLatestRestorableVersion() so that
it is transparent to clients.
2020-03-20 20:15:08 -07:00
e15015ee6c
Add mutation log version names
...
I.e., BACKUP_AGENT_MLOG_VERSION for 2001 and PARTITIONED_MLOG_VERSION for 4110.
2020-03-20 20:13:38 -07:00
924d335aa7
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# flow/Knobs.cpp
# flow/Knobs.h
2020-02-25 18:25:19 -08:00
d3bca19960
backup should also submit on the first proxy for similar reasons to DR
2020-02-25 15:57:32 -08:00
e71fe44ee3
Merge branch 'master' into features/icc
2020-02-08 21:33:02 -08:00
d5849af5c0
Address review comments
2020-02-05 10:33:51 -08:00
d09e07f1f5
Merge remote-tracking branch 'upstream/master' into features/icc
2020-02-04 10:26:18 -08:00
52c6737411
Rename backupLoggingEnabled as backupWorkerEnabled
...
To highlight the changes for 7.0 backup changes. By default,
backup_worker_enabled flag is set for 7.0 version.
2020-02-04 10:09:16 -08:00
0db03f1d3c
Use backup_logging_enabled flag
...
The default is to enable new backup workers. Users can disable this flag to
turn off the backup worker feature.
2020-02-03 20:03:22 -08:00
38aa1903fd
Add a DB configuration option for backup workers
...
Right now, the default is to keep the old backup behavior, i.e., do NOT use
backup workers. Specifically, if BackupType is not set (or is set to default),
the master will not recruit backup workers and will not add pseudo locality for
backup workers.
The StartFullBackupTaskFunc is updated to check if backup worker is enabled.
Only when it is not enabled, starting a backup will wait on all backup workers
to be started.
2020-01-31 19:29:09 -08:00
f7956cfbfc
Clear backup UID from backupStartedKey when finish/abort backups
...
Clearing this key signals backup workers that backup is no longer needed. When
no backup is going on, the backup workers switch to the NOOP state.
2020-01-31 19:29:09 -08:00
19ef7f6bdb
Skip watch of backup task's started key if it's already set
...
The backup task may be restarted multiple times so the started key for the
backup task may already be set. In this case, the wait on watch should be
skipped.
2020-01-31 19:29:09 -08:00
f8342f0884
Add keepRunning for start backup transaction
...
TaskBucket::keepRunning() needs to be called in backup transactions to be sure
that the task has not been cancelled. If so, the task is cancelled. Otherwise,
the task can continue run, causing multiple runs of the same task.
Another subtle issue is that the beginVersion is persisted on backupStartedKey.
So while reading it back from that key, we should set task's beginVersion with
the value persisted earlier.
2020-01-31 19:29:09 -08:00
5a602f58e8
Start backup with a wait on all backup workers running
...
This wait is to make sure that backup workers are already saving mutations so
that no mutations are missed. The idea is that the CLI sets a "backupStartedKey"
in the database and waits for allWorkerStarted() key of the backup to be set.
Backup workers monitor the changes to the "backupStartedKey" and start logging
mutations. Additionally, backup worker for Tag(-2,0) monitors all other workers
have started (checking their saved progress version is larger than the backup's
start version), and then sets the allWorkerStarted() key for the backup.
2020-01-31 19:29:09 -08:00
7f7ec99170
Serialize and deserialize new backup files
...
The BackupWorker writes files that can be read by FileConverter. Move
StringRefReader to the header file for reuse in FileConverter.
2020-01-22 19:38:46 -08:00
7628d04fb9
Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2020-01-09 07:21:16 -08:00
e98b2a0d1c
FastRestore:Introduce RestoreAsset
2019-12-20 18:00:10 -08:00
4ff058e86b
Backup and DR layer status document generation now uses snapshot reads for all keys read to avoid unnecessary conflicts when read during a status update or cleanup transaction. Since many of the keys read use wrapper functions, all of the underlying functions in BackupAgentBase and its two implementations also required a snapshot mode argument. All snapshot arguments default to false to match the underlying FDB API get/getrange methods.
2019-12-19 00:29:35 -08:00
Meng Xu