e32a0fb742
Resolve conflicts
2021-01-14 20:13:29 +00:00
6d6b2843da
Merge remote-tracking branch 'upstream/release-6.2' into anoyes/merge-release-6.2
...
I'm including the conflict markers for now and I'll resolve them in a
subsequent commit
2021-01-14 17:20:11 +00:00
a860778b51
Add write buffering to BackupContainerLocalDirectory::BackupFile to greatly reduce the number of IAsyncFile::write() calls made when writing backup data. The buffer size is controlled by a knob.
2021-01-09 07:57:48 -08:00
d4e3e98957
Use new getRestoreSet in RestoreController
2020-09-17 17:56:36 -07:00
71309a79f9
Merge branch 'release-6.3' into range
2020-09-17 17:54:33 -07:00
8224e17a08
Integrate extended getRestoreSet API into fast restore
...
The extended getRestoreSet provides a much smaller set of backup files for small keyrange restore;
This commit integrate it into fast restore so that fast restore does not have to filter out unneeded backup files.
2020-09-15 09:38:43 -07:00
f9a5b727a6
Add comments questions and TODOs
2020-09-14 16:28:01 -07:00
5880bd9fa7
BackupContainer:Fix getSnapshotFileKeyRange_impl on blob http error
2020-09-09 08:47:43 -07:00
3728ed03dd
Resolve comments
2020-09-05 18:55:09 -07:00
1ad5e17458
add support for comparing original and current impls
2020-09-05 11:14:59 -07:00
b6c0299d09
Add help message in backup CLI for added options
2020-08-31 09:31:57 -07:00
33aa10b461
Minor optimizations
2020-08-29 20:10:45 -07:00
fd7198d874
Extend backup container interface to support query restorable files set by key ranges
2020-08-29 19:58:07 -07:00
ca9b1f5b34
Merge branch 'release-6.3' into mengxu/fr-sched-PR
...
Resolve conflict at BackupContainer.actor.cpp
2020-08-27 16:54:00 -07:00
369000a125
BackupContainer:Remove link to filename with random string
2020-08-26 15:55:36 -07:00
a2ab709a0c
BackupContainer:Use processId as the process filename
...
instead of using a randomly generated string which change every time
when a file is open.
Having too many files will trigger TOO_MANY_FILES error
2020-08-26 15:54:34 -07:00
6256bedf8d
BackupContainer:Use processId as the process filename
...
instead of using a randomly generated string which change every time
when a file is open.
Having too many files will trigger TOO_MANY_FILES error
2020-08-25 12:25:09 -07:00
d2cf700bd4
Fix compiler warnings
2020-07-28 18:30:26 +00:00
2ac7df8a18
Merge pull request #3563 from xumengpanda/mengxu/fr-filesize-PR
...
FastRestore:Add trace for file size and bc progress
2020-07-25 20:08:57 -07:00
99d8399f4e
FastRestore:Add trace for file size and bc progress
2020-07-25 19:12:25 -07:00
e1dedff7b3
Merge branch 'release-6.2' into release-6.3
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# cmake/ConfigureCompiler.cmake
# documentation/sphinx/source/downloads.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/FileBackupAgent.actor.cpp
# packaging/msi/FDBInstaller.wxs
# versions.target
2020-07-24 12:10:44 -07:00
6446b4c082
WIP
2020-07-09 22:02:43 +00:00
07a9a05683
FastRestore:Agent:Fix restore requests
2020-04-30 16:20:20 -07:00
6ee78aa3a4
Fix:Disable sanity check backup metadata file
...
Which can increase the false positive rate of TooManyFiles error
2020-04-30 16:16:14 -07:00
a8becb9027
Use calculated block size for range files in unit tests
2020-04-28 21:13:18 -07:00
ba261eda36
Fix a backup container unit test
...
Write a valid range file instead of random data so that checking its content is
fine.
2020-04-28 15:39:24 -07:00
59217ddf1e
Remove sanity check on metadata
...
The sanity check parses each range file to get the key range of each
range file.
The parsing incurs restore_unsupported_file_version error.
We need to include this sanity check before 6.3 release.
2020-04-23 14:59:58 -07:00
9bfc5bbea8
Check RestorableFileSet's key ranges in simulation
...
Ranges written in the manifest file should match with actual file content.
2020-04-21 12:55:40 -07:00
0938e45c6a
Set continuous version to invalidVersion when snapshot version is the target version
2020-04-20 22:26:42 -07:00
930b175c4c
Add range files' key ranges to RestorableFileSet
...
Also add continuous logs' begin and end version in RestorableFileSet.
2020-04-20 22:26:42 -07:00
a2b867c6f9
Fix a unit test failure
2020-04-20 22:26:42 -07:00
3063611355
Write range files' begin & end keys to manifest file
...
This information can be very useful in knowing the content in these files,
especially for restores.
2020-04-20 22:26:42 -07:00
4c66c8c377
Fix backup progress calculation
...
The oldest epoch the master gets can assume its begin version is 1, which can
be wrong. In this case, we use the saved backup progress to "true-up" the real
begin version.
2020-04-20 11:06:46 -07:00
db2cef844b
Write mutation log type as a backup property
...
This can solve the problem when listing log files returns empty results.
2020-04-09 09:29:24 -07:00
4d06e837dc
Remove getPartitionedRestoreSet() API
...
Use getRestoreSet() instead for both old and new partitioned logs.
2020-04-08 20:12:09 -07:00
fd9caa88a0
Remove isPartitionedBackup()
...
This is no longer needed, since describeBackup() figures this out.
2020-04-08 16:09:18 -07:00
0cf6013357
Refactor to remove describePartitionedBackup()
...
The backup container can figure out if partitioned logs are used by looking at
mutation logs, thus consolidating the API to a single describeBackup() as
before.
2020-04-08 15:59:37 -07:00
241f9c123e
Try to find continuous log ranges that cover snapshot begin version
...
Backup container can have mutation log files that are not continuous overall,
but contain a continuous range that cover the snapshots. So when determine the
continuous log ranges, try to find one that cover the first snapshot's begin
version.
2020-03-28 21:19:47 -07:00
3261eae8c1
Initialize contiguousLogEnd to the lowest possible version
...
If set to the first file's end version, then we may erroneously set to an
unrestorable version.
2020-03-28 17:38:02 -07:00
34745d71fa
Fix continuous end version
...
When scan from a known version, stop if logs are not continuous. However, if
scan from 0, we should reset minLogBegin to the next file and continue scan
from that file's begin version.
2020-03-28 17:38:02 -07:00
abcb458a44
Fix contiguousLogEnd calculation
...
The contiguous version should scan from the max of scanBegin version and the
minLogBegin version. Once we found a version that's larger, set it as the
contiguousLogEnd version.
2020-03-28 17:38:02 -07:00
32b0ba1822
Merge branch 'master' into mengxu/parallel-range-log-file-loading-PR
2020-03-27 12:13:47 -07:00
113d0fb48b
Remove incorrect assertion
2020-03-27 12:13:30 -07:00
772ab70aee
Add an option for fast restore to restore old backups
...
If "usePartitionedLogs" is set to false, then the workload uses old backups for
restore.
2020-03-26 13:04:00 -07:00
799f0b4b0e
Small code refactor
2020-03-20 20:15:09 -07:00
4c75c61f39
Fix duplicate file removal for subset version ranges
...
Partitioned logs can have strict subset version ranges, which was not properly
handled -- we used to assume overlapping only happens for the same begin
version.
2020-03-20 20:15:09 -07:00
c63493c34f
Allow overlapped versions in partitioned logs
...
The overlapping can only happens between two generations, where the known
committed version to recovery version is copied from old generation to the new
generation. Within a generation, there is no overlap.
The fix here is related to the calculation of continuous version ranges,
allowing the overlap to happen.
2020-03-20 20:15:09 -07:00
4f4ce93f8c
Remove debug print out
2020-03-20 20:15:09 -07:00
fe6b4a4398
Some correctness fixes
2020-03-20 20:15:08 -07:00
5ce9fc0e4c
Partitioned logs should be filtered after sorting by tag IDs
...
The default sorting by begin and end version doesn't work with duplicates
removal, as tags are also compared.
2020-03-20 20:15:08 -07:00
Andrew Noyes