594e8944ae
Move RestoreWorkerInterface into fdbserver
2021-05-30 11:51:47 -07:00
bb076115c9
Only enable backup worker when using partitioned logs
...
This addresses issue #4849 .
2021-05-21 16:40:29 -07:00
5c2d7b6080
Create RangeResult type alias
2021-05-03 13:14:16 -07:00
5e045bd21c
Move the read of logsOnly and inconsistentSnapshotOnly to a loop where other RestoreConfig vars are fetched.
2021-04-29 17:00:34 -07:00
44197644be
Reset transaction after a commit and before start a read.
2021-04-29 12:19:58 -07:00
2f3d70c084
Fix the logic of getting firstConsistentVersion.
...
First consistent version should be:
- In a logs-only restore, it is the begin version the user said to start applying logs for;
- In an inconsistent-snapshot-only restore, if all range files have the same version, then it is that version, otherwise unknown (use -1);
- If using both range files and logs, then it is the highest version of any range file in the RestoreSet’s ranges vector.
2021-04-27 11:27:57 -07:00
719f810676
Rename incrementalBackupOnly to onlyAppyMutationLogs in all restore configs and functions.
2021-04-26 12:30:46 -07:00
41ca11c3e5
Implement restoring an inconsistent snapshot as a real feature.
2021-04-22 13:53:37 -07:00
bc43fa99ac
Move commit to its own try loop.
2021-04-21 17:37:58 -07:00
b90f61d740
Move commit to its own try loop.
2021-04-21 15:50:25 -07:00
7c4b5b0337
Add first consistent version in restore status.
...
First consistent version is the max of versions in RestoreFileSet.
2021-04-21 14:32:13 -07:00
ba60f18ebf
Merge remote-tracking branch 'upstream/master' into backup-agent
2021-04-20 22:54:47 -07:00
81fbe9ceaa
Merge pull request #4684 from sfc-gh-satherton/restore-range-fixes
...
Restore target range handling bug fixes
2021-04-20 11:57:13 -07:00
75425b5a24
Merge pull request #4620 from RenxuanW/renxuan/first-pr
...
Control backup's initial snapshot interval via backup cmd argument.
2021-04-19 23:59:02 -07:00
3f54a4a6dc
Throw an error if an empty range set is passed to restore().
2021-04-19 21:52:38 -07:00
c2c9ca4362
Assert was incorrect. Restore ranges must begin with the restore prefix to remove.
2021-04-19 17:01:20 -07:00
ab4c5ff90e
For better readability
2021-04-19 14:06:50 -07:00
03c031a09d
Update getCurrentVersion_impl
...
- If the restore is in the running state, then the current version is the getApplyBeginVersion()
- If the restore is in the completed state, the current version is the restore target version which comes from the restoreVersion() property.
- If the restore is in any other state, the current version can be reported as -1 as you have done.
2021-04-19 13:43:51 -07:00
0378dc0a50
Report the current version in the restore status.
2021-04-14 22:19:39 -07:00
97b995fb4f
Update fdbclient/FileBackupAgent.actor.cpp
...
Co-authored-by: Trevor Clinkenbeard <trevor.clinkenbeard@snowflake.com>
2021-04-14 13:49:59 -07:00
a285d6019e
We cannot put 2 Future functions in the same wait if the second one uses the first's result.
...
Before this change:
20210414-180825-renxuan-7451fad7aed4f0c7 compressed=True data_size=22960315 duration=732 ended=146 fail=10 fail_fast=10 max_runs=100000 pass=46 priority=100 remaining=0 runtime=0:01:12 sanity=False started=147 stopped=20210414-180937 submitted=20210414-180825 timeout=5400 username=renxuan
After this change:
20210414-192849-renxuan-cbe0f71ad5c48286 compressed=True data_size=22959419 duration=4261266 ended=106778 fail=1 fail_fast=10 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=0:24:49 sanity=False started=106963 stopped=20210414-195338 submitted=20210414-192849 timeout=5400 username=renxuan
2021-04-14 13:10:56 -07:00
9737212e51
The default value of the first snapshot interval should be 0 rather than -1.
2021-04-14 10:56:42 -07:00
a0430536f1
Remove knob BACKUP_INIT_SNAPSHOT_INTERVAL_SEC.
2021-04-14 10:41:41 -07:00
ebf37594f7
Change initialSnapshotIntervalSeconds from knob to a backup argument.
2021-04-13 19:22:13 -07:00
c8b27e71c5
Revert TraceEvent
...
We've found the problem (issue #4640 ), so we no longer need the TraceEvent.
2021-04-12 15:03:47 -07:00
dc00d99626
Log FileBackupLogRangeStart before calling getLogRanges() in .
...
It will tell us if or why this function is legitimately trying to use too much ram. getLogRange() should normally return about 20 items in the result. If the inputs are trash, it could return far more.
If it isn’t the case, then there’s something else wrong that has corrupted something such that when we try to allocate memory.
2021-04-09 15:02:45 -07:00
fadc9cccee
Use knob RESTORE_IGNORE_LOG_FILES in restore.
...
Rename IGNORE_LOG_FILES to RESTORE_IGNORE_LOG_FILES. Also, this knob should be used in regular restore, not parallel restore.
2021-04-07 10:05:56 -07:00
1b6ad42db8
Use a knob to completely ignore log files
2021-04-06 19:07:01 -07:00
edb3dd4414
Control backup's initial snapshot interval via knob.
2021-04-05 18:25:18 -07:00
2b4744ad8a
Reword comments.
2021-03-29 13:43:43 -07:00
55ef40864f
Add comments for touched functions
2021-03-29 10:09:07 -07:00
1a0ef39e5d
Fix keyRangesFilter with logsOnly flag
...
Remove the Sev 40 trace event.
2021-03-27 21:48:27 -07:00
a9d8806828
Use the restored range in the actual restore
2021-03-25 10:07:25 -07:00
720c0b3a4f
Added const.
...
Co-authored-by: Trevor Clinkenbeard <trevor.clinkenbeard@snowflake.com>
2021-03-24 14:36:55 -07:00
a6136ffc62
Applied clang-format to recent changes.
2021-03-24 06:51:08 -07:00
17a19021fb
Added mutation batch filtering during restore, to the extent possible where each restore mutation log task only sees one block of data (usually 1MB) which can contain incomplete and therefore unfilterable mutation sets.
2021-03-24 06:38:50 -07:00
3751ecd11f
Merge pull request #4503 from sfc-gh-ajbeamon/merge-release-6.3-into-master
...
Final merge of release-6.3 into master
2021-03-16 09:29:05 -07:00
25c4880ebe
Merge branch 'release-6.3' into merge-release-6.3-into-master (temporarily discard all changes to BackupContainer.actor.cpp)
...
# Conflicts:
# fdbclient/BackupContainer.actor.cpp
# fdbserver/Knobs.h
2021-03-15 16:41:04 -07:00
766b9d24d6
Merge branch 'master' into remove-invalid-backup-status-fields
2021-03-14 18:57:03 -07:00
cfbaaa6001
Merge remote-tracking branch 'origin/release-6.2' into release-6.3
2021-03-10 14:30:47 -07:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
f81d4eb6d3
Remove invalid backup status fields
2021-03-08 15:48:46 -08:00
318b862aa3
Apply clang-format to backup changes.
2021-03-08 01:49:29 -08:00
8a4aca3f47
Bug fix: In simulation only, RangeFileWriter was not waiting on a write future in one location.
2021-03-07 22:19:04 -08:00
8a8c488ede
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-05 18:13:38 -06:00
aaf0a9aa7b
Merge branch 'release-6.3' into merge-release-6.3-into-master
...
# Conflicts:
# build/docker-compose.yaml
# cmake/ConfigureCompiler.cmake
# fdbclient/FileBackupAgent.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/IAsyncFile.h
# fdbrpc/IRateControl.h
# fdbrpc/simulator.h
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbservice/ServiceBase.cpp
2021-02-08 12:58:34 -08:00
3fc8870ce3
winbuild: fix duplicate definition of Codec<ERestoreState>::pack and unpack in Debug build
...
- defined twice in FileBackupAgent.actor.cpp and BackupAgent.actor.h
- only fails when building in Debug ?
2021-01-30 19:03:25 +01:00
cec050a1f8
Apply clang-tidy's modernize-use-override fix
2021-01-26 02:04:12 +00:00
4669f837fa
Add uses of makeReference
2020-11-07 22:10:18 -08:00
51db9a7e0a
add static method to access backup pause key instead of constructing it manually
2020-11-06 14:03:29 -05:00
c1d88b39ab
Merge remote-tracking branch 'origin/master' into add-upgrade-and-backup-restore-test
2020-10-17 11:35:12 -07:00
a1e868a569
Merge pull request #3862 from sfc-gh-tclinkenbeard/use-override-more
...
Add uses of override keyword, remove unnecessary uses of virtual
2020-10-14 15:06:45 -07:00
ccf2ca73f3
Fix Windows CMake build
2020-10-08 23:58:58 -07:00
a9607bdcec
Explicitly seal classes that inherit but aren't inherited from
2020-10-07 21:58:24 -07:00
a3de8c333b
Use override where applicable in fdbclient
2020-10-07 16:57:38 -07:00
7218586f2f
adjust test spec and backup describe
2020-10-06 17:03:09 -04:00
43ac7f21a4
Mark toString methods const
2020-10-04 23:27:10 -07:00
8ae0679112
Remove unnecessary separation of declaration and definition of version field
2020-10-04 23:27:05 -07:00
6326de052b
Remove unused LogFileWriter::FFs
2020-10-02 17:03:08 -07:00
8da8c6ff0d
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-30 14:25:29 -04:00
44cd3b0999
add stopBackup to incrementalBackup workload
2020-09-30 14:24:52 -04:00
cfaa207f88
fix spacing and newlines at ends of files
2020-09-28 16:32:20 -04:00
3aa4070adb
remove trace events used for debugging
2020-09-28 16:27:30 -04:00
1fb11858e6
Fix failure caused by merge
2020-09-27 11:03:16 -07:00
bbc7ce581e
Resolve conflicts merging from 6.3 to master
2020-09-25 16:08:31 -07:00
6ed330572e
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-24 12:09:17 -04:00
d13a9d3c91
Make EBackupState an enum class
2020-09-23 15:14:09 -07:00
69580593dd
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-23 15:35:05 -04:00
9b7d75b6dd
WIP
2020-09-21 13:41:35 -04:00
a3c861b59d
Merge pull request #3777 from apple/release-6.3
...
Merge Release 6.3 into master
2020-09-14 10:32:50 -07:00
f10e9ea679
FastRestoreApplier:Add write conflict range
2020-09-09 12:12:14 -07:00
e69ee87f33
Refactor blob failure to simulateBlobFailure function
2020-09-09 10:54:21 -07:00
20733ae1eb
Simulate throwing blob http error immediately after read blob file
2020-09-09 06:47:19 -07:00
cffb54ba0b
Buggify decodeLogFileBlock with blob http error
2020-09-09 06:44:02 -07:00
f3986d201d
Buggify backup decodeRangeFileBlock with blob connection error
2020-09-09 06:40:41 -07:00
8ac3b21459
changed occurrences of getOrThrow to get
2020-09-02 13:45:15 -04:00
00c77ba2b4
Added beginVersion cmd line option and addressed code review comments
2020-08-28 14:29:22 -04:00
0270ee13c8
change targetversion from maxlogend to contiguouslogend. also adjusted test workload and specs
2020-08-28 12:46:34 -04:00
35aa1034cd
added new test spec and workload to verify correctness of incremental backup and restore
2020-08-19 13:45:51 -04:00
7dce3a9187
fixed issue with mutations not applying and allow backup to non-empty db
2020-08-11 15:39:21 -04:00
21635f8a28
update backup restore for local testing
2020-08-04 15:48:43 -04:00
fbdbe0f052
adjusted locations in which restore pipeline is modified
2020-07-28 12:39:35 -04:00
2d09d2f68e
re-ordered parameters to fit existing test workloads
2020-07-28 12:39:35 -04:00
71a1b2cb0d
first attempt at making restore pipeline incremental only
2020-07-28 12:39:35 -04:00
3c62651ad6
initial commit to introduce incremental backup only (ignore base snapshot)
2020-07-28 12:39:35 -04:00
b2a3b4fd83
Merge branch 'master' into mengxu/merge-6.3-PR
2020-07-20 11:34:18 -07:00
2e5ae8eed5
Change iterator name to small case
2020-07-10 10:56:41 -07:00
941cf7b03b
Fix const return warning
2020-07-09 22:17:54 +00:00
ab3de8700e
Rename fastRestoreAgent to fastRestoreTool
2020-07-03 00:16:39 +09:00
c13ba7aeb4
FastRestore:Cosmetic code style improvement
...
Also mute some trace events
2020-06-24 21:25:37 -07:00
94b1e17a96
FastRestore:Fix:Avoid reset txn backoff time in sanity check
2020-06-24 21:10:47 -07:00
d2b71e77c2
FastRestore:Fix:checkDatabaseLock tr should not reset
2020-06-24 20:21:49 -07:00
b69c36f0d6
transformDatabaseContents:cosmetic change
2020-06-23 19:27:58 -07:00
198696bc1e
Move transformRestoredDatabase from server to client
...
AtomicRestore workload turns out to rely on the FileBackupAgent
client. Keeping transformRestoredDatabase in server makes linking harder.
2020-06-23 15:48:43 -07:00
6eb8dc58b6
FastRestore:addPrefixTest:Fix calling transformRestoredDatabase
2020-06-23 15:02:38 -07:00
84a640e4ee
AtomicRestore:Add addPrefix test for fast restore
2020-06-23 14:52:27 -07:00
4e27fd34e5
Refactor transformDatabaseContents into RestoreCommon
...
Prepare to enable addPrefix for atomicRestore
2020-06-23 14:33:13 -07:00
2fcef90cbc
FastRestore:Add addPrefix and removePrefix feature
2020-06-18 11:58:42 -07:00
d7a8e554ed
FastRestore:Test for addPrefix and removePrefix
...
Add the skeleton code to test;
addPrefix and removePrefix are all empty for now.
2020-06-17 20:36:53 -07:00
164814cd23
Merge branch 'release-6.3' into mengxu/fr-addPrefix-PR
...
Also remove the unnecessary try catch block for unlockDatabase() in
parallelRestoreFinish actor
2020-06-09 22:04:19 -07:00
sfc-gh-tclinkenbeard