df064ac922
Merge pull request #3321 from xumengpanda/mengxu/fr-restore-ranges-PR
...
Fast Restore: Support restoring sub ranges in the framework
2020-06-09 21:01:39 -07:00
dd8af487ec
FileBackupAgent:parallelRestoreFinish:verify if operation_cancelled error ever happens
2020-06-09 17:08:57 -07:00
8533daac50
parallelRestoreFinish:fix used_during_commit error
2020-06-09 15:40:24 -07:00
e5e4cb5c90
parallelRestoreFinish:separate check restore finished and clear restoreRequestDoneKey
2020-06-09 11:32:20 -07:00
d85dc5a4d3
FastRestore:Only clear ranges that will be restored
...
Instead of clearning the entire normal key space.
This commit also removes some unnecessary tr->reset() which can invalid the txn backoff time.
2020-06-08 22:41:49 -07:00
e7fdd614fe
Remove assert on error_code_key_not_found in previous commit
...
Correctnss shows the assert does not hold in all scenarios
2020-06-08 19:02:56 -07:00
5022566b35
Validate if key_not_found error ever happens
2020-06-08 16:59:00 -07:00
a833724322
Merge pull request #3078 from xumengpanda/mengxu/fr-circus-stall-PR
...
Performant restore: Various improvements based on circus test
2020-05-07 20:07:23 -07:00
7b5d43da9c
FastRestore:Remove unused field in RestoreRequest
2020-05-03 20:59:47 -07:00
494d1d3518
Fixed unused variable warning in atomicParallelRestore
2020-05-01 18:14:51 -07:00
a998fa1e1a
Merge pull request #3053 from xumengpanda/mengxu/fr-bc-fix-PR
...
Fix performant restore bug introduced in PR 3041
2020-04-29 17:09:26 -07:00
3f510d0653
Merge pull request #3036 from jzhou77/backup-cmd
...
Several bug fixes for new backups
2020-04-29 16:34:59 -07:00
9da76c35ea
Fix a memory corruption error
...
The backup container URL should be Key instead of KeyRef.
2020-04-29 15:55:34 -07:00
33dedaab35
FastRestore:Fix submitParallelRestore when targetVersion is unset
2020-04-29 15:44:54 -07:00
364142d02c
Fix backupWorkerEnabled flag not set bug for first backup
...
This can cause restore failures because the latestLogEndVersion could be wrong.
2020-04-28 15:39:24 -07:00
7d59e53349
Consolidate makePadding()
2020-04-28 15:39:23 -07:00
f5e8345496
FastRestoreAgent:Use atomicParallelRestore to kick off restore
...
Replace the handcrafted version with atomicParallelRestore actor
which is simulation tested
2020-04-27 22:15:00 -07:00
3063611355
Write range files' begin & end keys to manifest file
...
This information can be very useful in knowing the content in these files,
especially for restores.
2020-04-20 22:26:42 -07:00
60407bdee3
Use LiteralStringRef for backup paused key
2020-04-07 16:02:25 -07:00
9fb3fb9d82
Add pause/resume for new backups
...
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
2020-04-06 14:29:46 -07:00
efbd01894d
Really fix the stuck of backup
2020-03-28 17:38:02 -07:00
f0d0c863e6
Attempt to fix
2020-03-28 17:38:02 -07:00
78f74ed6fe
Skip copying inside BackupLogsDispatchTask
2020-03-28 17:38:02 -07:00
a2b8be5dc6
Switch off old mutation logging on proxies for new backups
...
Specifically, stop writing LogRange mutation for new backup jobs.
2020-03-28 17:38:02 -07:00
9a9af7d8a8
Add more trace event details on partitioned log
2020-03-26 13:57:31 -07:00
feedab02a0
Merge pull request #2855 from xumengpanda/mengxu/fr-api-atomicrestore-PR
...
Add ApiCorrectnessAtomicRestore workload for the new performant restore
2020-03-25 18:05:26 -07:00
495afe2e0b
Improve how to wati for restore to finish
...
Remove default parameter for atomicRestore as suggested in review.
2020-03-25 13:54:21 -07:00
1ba11dc74b
Apply clang format
2020-03-25 11:20:17 -07:00
120272f025
Change unlockDB from RestoreMaster to Agent
2020-03-25 11:04:49 -07:00
130b91c820
Fix segmentation fault in submitParallelRestore
2020-03-24 23:52:34 -07:00
ca8966a28b
Move lockDB into submitRestore request from restore worker
...
AtomicRestore needs to lock DB before we start the restore worker.
So we cannot lock DB in restore worker with a different randomUID.
2020-03-24 23:39:35 -07:00
e59becdec8
Refactor atomicRestore for atomicParallelRestore
2020-03-24 22:37:12 -07:00
241c2703c8
Fix atomicParallelRestore interface
2020-03-24 17:00:55 -07:00
80d62f3cb8
Fix:Add atomicParallelRestore to header
2020-03-24 16:28:08 -07:00
ffb0a439ea
atomicParallelRestore should not return a version
2020-03-24 16:05:09 -07:00
b173929316
Add atomicParallelRestore to AtomicRestore workload
2020-03-24 15:58:49 -07:00
01921c3953
Move FileBackupAgent parallel restore impl to FileBackupAgentImpl
...
For consistency with existing code, and
state variable can only occur in ACTOR.
2020-03-24 15:11:34 -07:00
81f7181c9e
Refactor submitParallelRestore function into FileBackupAgent
2020-03-24 14:44:55 -07:00
5584884c12
Refactor parallelRestoreFinish function into FileBackupAgent
2020-03-24 14:15:15 -07:00
4e477f5489
Copy atomicRestore to atomicParallelRestore
2020-03-24 13:59:35 -07:00
e1737fc644
Skip setting backupStartedKey if using old mutation logs
...
For old submitBackup(), where partitionedLog is false, do not set the
backupStartedKey in BackupConfig, which signals backup workers to skip these
backups.
2020-03-20 20:15:09 -07:00
38def426f4
Add a flag to submitBackup for partitioned log
...
This is to distinguish with old workloads so that they can work in simulation.
2020-03-20 20:15:08 -07:00
e9287407d6
Backup worker updates latest log versions in BackupConfig
...
If backup worker is enabled, the current epoch's worker of tag (-2,0) will be
responsible for monitoring the backup progress of all workers and update the
BackupConfig with the latest saved log version, which is the minimum version
of all tags.
This change has been incorporated in the getLatestRestorableVersion() so that
it is transparent to clients.
2020-03-20 20:15:08 -07:00
e15015ee6c
Add mutation log version names
...
I.e., BACKUP_AGENT_MLOG_VERSION for 2001 and PARTITIONED_MLOG_VERSION for 4110.
2020-03-20 20:13:38 -07:00
924d335aa7
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# flow/Knobs.cpp
# flow/Knobs.h
2020-02-25 18:25:19 -08:00
d3bca19960
backup should also submit on the first proxy for similar reasons to DR
2020-02-25 15:57:32 -08:00
e71fe44ee3
Merge branch 'master' into features/icc
2020-02-08 21:33:02 -08:00
d5849af5c0
Address review comments
2020-02-05 10:33:51 -08:00
d09e07f1f5
Merge remote-tracking branch 'upstream/master' into features/icc
2020-02-04 10:26:18 -08:00
52c6737411
Rename backupLoggingEnabled as backupWorkerEnabled
...
To highlight the changes for 7.0 backup changes. By default,
backup_worker_enabled flag is set for 7.0 version.
2020-02-04 10:09:16 -08:00
0db03f1d3c
Use backup_logging_enabled flag
...
The default is to enable new backup workers. Users can disable this flag to
turn off the backup worker feature.
2020-02-03 20:03:22 -08:00
38aa1903fd
Add a DB configuration option for backup workers
...
Right now, the default is to keep the old backup behavior, i.e., do NOT use
backup workers. Specifically, if BackupType is not set (or is set to default),
the master will not recruit backup workers and will not add pseudo locality for
backup workers.
The StartFullBackupTaskFunc is updated to check if backup worker is enabled.
Only when it is not enabled, starting a backup will wait on all backup workers
to be started.
2020-01-31 19:29:09 -08:00
f7956cfbfc
Clear backup UID from backupStartedKey when finish/abort backups
...
Clearing this key signals backup workers that backup is no longer needed. When
no backup is going on, the backup workers switch to the NOOP state.
2020-01-31 19:29:09 -08:00
19ef7f6bdb
Skip watch of backup task's started key if it's already set
...
The backup task may be restarted multiple times so the started key for the
backup task may already be set. In this case, the wait on watch should be
skipped.
2020-01-31 19:29:09 -08:00
f8342f0884
Add keepRunning for start backup transaction
...
TaskBucket::keepRunning() needs to be called in backup transactions to be sure
that the task has not been cancelled. If so, the task is cancelled. Otherwise,
the task can continue run, causing multiple runs of the same task.
Another subtle issue is that the beginVersion is persisted on backupStartedKey.
So while reading it back from that key, we should set task's beginVersion with
the value persisted earlier.
2020-01-31 19:29:09 -08:00
5a602f58e8
Start backup with a wait on all backup workers running
...
This wait is to make sure that backup workers are already saving mutations so
that no mutations are missed. The idea is that the CLI sets a "backupStartedKey"
in the database and waits for allWorkerStarted() key of the backup to be set.
Backup workers monitor the changes to the "backupStartedKey" and start logging
mutations. Additionally, backup worker for Tag(-2,0) monitors all other workers
have started (checking their saved progress version is larger than the backup's
start version), and then sets the allWorkerStarted() key for the backup.
2020-01-31 19:29:09 -08:00
7f7ec99170
Serialize and deserialize new backup files
...
The BackupWorker writes files that can be read by FileConverter. Move
StringRefReader to the header file for reuse in FileConverter.
2020-01-22 19:38:46 -08:00
7628d04fb9
Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2020-01-09 07:21:16 -08:00
e98b2a0d1c
FastRestore:Introduce RestoreAsset
2019-12-20 18:00:10 -08:00
4ff058e86b
Backup and DR layer status document generation now uses snapshot reads for all keys read to avoid unnecessary conflicts when read during a status update or cleanup transaction. Since many of the keys read use wrapper functions, all of the underlying functions in BackupAgentBase and its two implementations also required a snapshot mode argument. All snapshot arguments default to false to match the underlying FDB API get/getrange methods.
2019-12-19 00:29:35 -08:00
f3ed3f462b
Added code comments in backup log range task.
2019-12-03 09:44:15 -08:00
b7b5d2ead3
Remove several nonsensical const uses
...
These seem to be all the ones that clang's -Wignored-qualifiers
complains about
2019-10-26 14:30:34 -07:00
688940b685
merge 6.2 into master
2019-10-21 11:43:46 -07:00
628b4e0220
added a warning if multiple log ranges exist for the same range
2019-10-02 17:06:19 -07:00
d0147e5e5d
Merge branch 'release-6.2' into mengxu/merge-release620-to-master-v3
...
Resolved Conflicts:
documentation/sphinx/source/release-notes.rst
fdbserver/DataDistribution.actor.cpp
versions.target
2019-10-02 13:22:56 -07:00
9ec9f41d34
backup and DR would not share mutations if they were started on different versions of FDB
2019-10-01 18:52:07 -07:00
4d659662b8
made cleanup handle retries better
2019-09-30 12:44:20 -07:00
ef01ad2ed8
optimized log range clearing to clear everything for each possible hash (256 clears) if that would be more efficient than one clear per second that has elapsed
...
aborting a DR without the —cleanup flag will still attempt to cleanup for 30 seconds before giving up
added a cleanup command to fdbbackup which can remove mutations from orphaned DRs which were stopped without the —cleanup flag
2019-09-27 18:32:27 -07:00
d160810662
FastRestore:Resolve review comments
2019-09-04 16:48:43 -07:00
2602cb3591
FastRestore:Rename RestoreConfig to RestoreConfigFR to fix link problem in windows
...
Because the current restore has defined RestoreConfig, windows linker complains.
This commit rename the RestoreConfig used in FastRestore as RestoreConfigFR.
2019-08-02 23:00:12 -07:00
3b54363780
FastRestore:Apply Clang-format
2019-08-01 18:09:12 -07:00
45083edf74
Merge branch 'master' into mengxu/performant-restore-PR
...
Fix conflicts as well.
2019-07-25 10:46:11 -07:00
844dd60202
FDB compiling with intel compiler
2019-06-20 09:29:01 -07:00
477fd152c0
FastRestore:Refactor code
...
1) Use the runRYWTransaction for simple DB access
2) Replace some printf with TraceEvent
3) Remove printf not used in debugging
4) Avoid wait inside the condition in loop-choose-when for
the core routine of restore worker, loader and applier.
5) Rename Restore.actor.cpp to RestoreWorker.actor.cpp since
the file only has functionalities related to restore worker.
Passed correctness test
2019-06-04 11:22:47 -07:00
603721e125
Merge branch 'master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/genericactors.actor.cpp
# fdbrpc/sim2.actor.cpp
# fdbserver/DiskQueue.actor.cpp
# fdbserver/workloads/BulkSetup.actor.h
# flow/ActorCollection.actor.cpp
# flow/Net2.actor.cpp
# flow/Trace.cpp
# flow/flow.cpp
2019-05-23 08:35:47 -07:00
f4fbaac6b0
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-05-19 10:27:59 -07:00
e495afdc6e
Removed unnecessary c_str().
2019-05-16 17:26:37 -07:00
7b8e140788
Added UID field to fdbbackup status --json output.
2019-05-16 17:23:33 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
b5cbffc1b8
fdbclient: fix some print/scan format warnings
...
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-06 13:35:29 -07:00
529ce66b6c
Merge branch 'apple/master' into mengxu/performant-restore-PR
2019-04-18 18:02:45 -07:00
781b6ece77
Fix OPEN_FOR_IDE -Wunused-variable warnings
...
CC #1255 , #1173
2019-04-16 15:28:01 -07:00
6207d724f8
Fix all -Wunused-variable warnings
2019-04-15 18:13:00 -07:00
1c16f87a4e
Remove trace-calls to printable (in non-workloads)
2019-04-05 13:12:19 -07:00
c4a8a80d6f
Merge branch 'apple/master' into mengxu/performant-restore-PR
2019-04-04 22:51:00 -07:00
4ed8e9c16f
FastRestore: Comment out fprintf stderr
...
We used fprintf(stderr,) to make sure the message is flushed out.
When we run correctness, we should comment the unnecessary stderr
to avoid false positive errors.
2019-04-01 18:35:54 -07:00
068ba2e082
FastRestore: Rename RestoreFile to RestoreFileFR to avoid weird running error
...
When two struct have the same name but never used in the same scope,
the compiler will NOT report any error in compilation, but
the program will arbitrarily choose one of the struct at the linker time,
and experience weird error in running time. The runtime error is caused by the
corrrupted memory when we assign a struct content to a different struct type.
2019-04-01 18:23:38 -07:00
bae815a777
Bug fix, starting a restore on a tag already in-use would spinloop forever and eventually run out of memory.
2019-04-01 15:00:24 -07:00
70d7c289f4
Merge branch 'master' into mengxu/restore/parallel-v7
2019-03-30 22:13:10 -07:00
589fb76826
FastRestore:Attempt to fix old restore
2019-03-30 15:19:30 -07:00
5e9a6edfe6
FastRestore:bug fix: Lock DB successfully
2019-03-29 13:31:38 -07:00
c6edcc7f06
Added schema version string to backup JSON status docs. Bug fix in backup status JSON, the document was being created outside the transaction retry loop so retries would combine partial element sets across all tries into the result.
2019-03-14 02:10:14 -07:00
8aab719c22
Merge branch 'master' into feature-backup-json
2019-03-12 18:23:16 -07:00
a25e224cda
Merge pull request #1213 from etschannen/feature-metadata-version
...
Added a metadata version key
2019-03-12 13:36:33 -07:00
bc0b2aa040
Merge branch 'release-6.0' of https://github.com/apple/foundationdb
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BlobStore.actor.cpp
2019-03-12 04:49:12 -07:00
f0eae0295f
Merge branch 'master' of https://github.com/apple/foundationdb into feature-backup-json
2019-03-12 03:35:03 -07:00
c1745b9061
Bug fix, backup snapshot dispatch shardsBehind calculation would divide by zero if the snapshot scheduled interval is 0.
2019-03-11 19:43:59 -07:00
f0024c876e
Changed backup paused JSON field name to be more precise and match the human readable status version.
2019-03-11 01:42:19 -07:00
f2953db7d8
Added updating of backup snapshot shards behind in snapshot dispatcher so status can determine if a snapshot is lagging the configured speed.
2019-03-11 01:25:51 -07:00
2627bcd35e
Merge branch 'master' into feature-metadata-version
2019-03-10 21:13:28 -07:00
023bbb566f
Renamed backup state enums for clarity, added backup state names. Changed Epochs to EpochSeconds in backup JSON along with some other renaming/moving of fields, and added information about snapshot dispatch. Changed timestamp format for input/output in all backup/restore contexts to be a fully qualified time with timezone offset. Added information about the last snapshot dispatch to backup config and status (not yet populated).
2019-03-10 16:00:01 -07:00
b9acc9a0e8
Code refactor to fix windows msvc14 compiler errors.
2019-03-08 15:13:11 -08:00
06c11a316d
Normalized timestamp to text format across backup and restore tooling. Added epochs field to JSON objects describing versions and timestamps in backup status and describe output, renamed some fields for clarity.
2019-03-06 22:34:25 -08:00
1399aee532
Added --json option to fdbbackup status.
2019-03-06 21:32:46 -08:00
43c290b546
Address review comments.
2019-03-06 17:39:29 -08:00
7778112f6a
Bug fix, restore was using the destination cluster to look up timestamps when printing the backup description instead of (optionally) the original cluster which generated the backup. Made missing cluster file errors more clear.
2019-03-06 02:45:55 -08:00
21f55e1878
Merge pull request #1190 from bnamasivayam/restore-multiple-ranges
...
Add support for restoring multiple ranges.
2019-03-05 10:15:55 -08:00
f6559bbb11
Added help and documentation for fdbbackup modify. Add BackupUID and more parseable BackupURL to fdbbackup status output.
2019-03-05 05:13:48 -08:00
d3377722d5
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 04:00:11 -08:00
5231ee6da0
make sure the version of a cluster is larger than the restore version, so future version stamps are larger than restored version stamps
2019-03-03 21:34:21 -08:00
2168b14834
Merge branch 'master' into feature-metadata-version
2019-02-28 17:45:55 -08:00
3da85f3acd
implemented the \xff/metadataVersion key, which can be used by layers to help them cheaply cache metadata and know when their cache is invalid
2019-02-28 17:45:00 -08:00
7d287c6999
Merge branch 'release-6.0'
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
2019-02-28 14:01:00 -08:00
887856b6b0
Bug fix in AsyncFileReadAhead where a file size that is an integer multiple of the read chunk size will cause a crash when reading the file's final block. BackupContainerLocalDirectory now uses AsyncFileReadAhead in simulation to get simulation coverage of that class, and FileBackup will generate file sizes which expose the bug.
2019-02-28 00:22:38 -08:00
7eba50b086
Add support for restoring multiple ranges.
2019-02-25 18:00:28 -08:00
4be90c91d6
Moved THIS and THIS_ADDR to actorcompiler.h
2019-02-19 15:16:59 -08:00
f3cd9f15ca
Update fdbclient/FileBackupAgent.actor.cpp
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-02-19 15:16:59 -08:00
82cfb94f4a
Update fdbclient/FileBackupAgent.actor.cpp
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-02-19 15:16:59 -08:00
f79a9594c1
Several bugfixes to make fdb build on non-ide
2019-02-19 15:16:59 -08:00
7fa7db3251
Minor bug fix - fdbclient now compiling
2019-02-19 15:16:59 -08:00
c1592b4c3a
Several minor fixes within fdbclient
2019-02-19 15:16:59 -08:00
70e089e791
Define THIS_ADDR for use with intellisense
...
this is undefined within ACTORs if compiled with
intellisense. We use a macro here to make sure just
replaces this with 0
2019-02-19 15:16:59 -08:00
78dd80ea8a
Proper fwd decl in BackupAgent
...
Also BackupAgent.h -> BackupAgent.actor.h
2019-02-19 15:16:59 -08:00
3cb2391b58
use proper fwd declarations in ManagementAPI
...
Also ManagementAPI.h -> ManagementAPI.actor.h
2019-02-19 15:16:59 -08:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
0750dc0418
Change store from (Future, T&) to (T&, Future).
...
LHS = RHS, and the name of what's being modified is easier to find.
2019-02-04 18:04:22 -08:00
684a22a52b
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BackupContainer.actor.cpp
# fdbclient/HTTP.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/BackupCorrectness.actor.cpp
# versions.target
2019-01-09 16:14:46 -08:00
604ad062d5
Updated backup correctness test to new behavior. WaitBackup() can now return the UID and BackupContainer atomically with the status code for a backup tag.
2019-01-08 18:12:15 -08:00
d2d68aa171
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# versions.target
2018-12-03 18:26:52 -08:00
e974ee367d
Bug fix, backups in progress during upgrade to 6.0.16 or later do not have have range file count initialized so reading it must default to 0.
2018-12-02 22:50:26 -08:00
1f3b6e8bdf
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/BackupContainer.actor.cpp
# fdbclient/BlobStore.actor.cpp
# versions.target
2018-11-27 14:41:46 -08:00
ec9410492d
Changed backup folder scheme to use a much smaller number of unique folders for kv range files, which will speed up list and expire operations. The old scheme is still readable but will no longer be written except in simulation in order to test backward compatibility.
2018-11-23 05:23:56 -08:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
0217aed74c
Merge branch 'release-6.0'
...
# Conflicts:
# bindings/go/README.md
# documentation/sphinx/source/release-notes.rst
# fdbserver/MasterProxyServer.actor.cpp
# versions.target
2018-10-15 18:38:51 -07:00
770b4b858b
Minor (and asymptomatic) bug fixes where ints or enumerations are being passed as the bool snapshot argument instead of as the default value which was intended.
2018-10-12 15:03:21 -07:00
3922e477a5
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/LogSystemDiskQueueAdapter.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
2018-10-03 16:57:18 -07:00
3f86905ea7
fix: restore did not take into account that the end version of a log file does not exist in that file. This resulted in restores done at the same version a snapshot completes to not apply the mutations at that final version.
2018-09-21 11:48:28 -07:00
e1fafbf259
Add some missing actorcompiler.h includes
2018-08-22 09:40:45 -07:00
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
1c29275672
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-01 14:30:57 -07:00
99c9958db7
Some more trace event normalization
2018-06-08 13:57:00 -07:00
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
e82985aea2
fix: continue setting beginVersion so that versions between 5.2.0 and 5.2.2 do not crash when decoding tasks created by 5.2.3
2018-06-06 13:34:22 -07:00
4120062bb9
fix: backup initialized its begin version at 1 instead of the read version of the starting transaction
...
fix: erasing log ranges did not properly divide up work between transactions to prevent making transactions which were too large
2018-06-06 13:05:53 -07:00
c37b48308e
Fix the way to get tagName
2018-05-03 11:11:44 -07:00
e5320fa237
Merge pull request #292 from ajbeamon/printable-encode-backup-beginfile
...
Encode BeginFile TraceEvent detail with printable
2018-05-01 15:53:48 -07:00
f848c68b73
TraceEvents in FileBackupAgent that include BeginFile should be encoding them with printable because of the possible inclusion of the null byte. This causes the detail to terminate prematurely.
2018-05-01 15:35:48 -07:00
52cc592d5e
Abort 5.1 backup for 5.2 upgrade
2018-05-01 14:49:14 -07:00
8d7b9fd506
Merge branch 'release-5.2' of github.com:apple/foundationdb into optimize-backup-priorities
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
2018-03-23 12:10:39 -07:00
ede5cab192
Merge pull request #89 from yichic/share-log-mutations-5.2
...
Share log mutations 5.2
2018-03-19 12:01:26 -07:00
ec02e54f64
Refactor EraseLogData()
2018-03-19 11:56:01 -07:00
1f2602d2b3
Fix all review comments
2018-03-19 11:33:33 -07:00
d6559b144f
Share log mutations between backups and DRs which have the same backup range
2018-03-19 11:32:50 -07:00
c5bb3421a7
Optimized backup task priorities so that during the initial snapshot range tasks are prioritized and then afterwards log tasks are prioritized.
2018-03-13 21:49:25 -07:00
94e90454da
Fix line endings
2018-03-05 12:16:37 -08:00
066a9b7e55
Declare each state variable on its own line...
2018-03-02 16:30:13 -08:00
34715ec1fa
Improve fdbbackup status
...
Additional details added are
log bytes written
range bytes written
last snapshot version and its corresponding timestamp
last log version and its corresponding timestamp
If backup is supposed to stop at the next snapshot completion (stopWhenDone)
version/timestamp of snapshot start and expected end
Backup error reporting format changed to
Recent Errors (since latest restore point TIME ago)
<TIME> ago: <error description> on <task name>
Older Errors
<TIME> ago: <error description> on <task name>
TIME format:
<= 59 seconds ago
<= 59.99 minutes ago
<= 23.99 hours ago
N.NN days ago
2018-02-28 18:22:45 -08:00
e1162e9238
Merge remote-tracking branch 'upstream/release-5.1'
2018-02-22 11:16:12 -08:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
aadc06de99
Merge remote-tracking branch 'upstream/release-5.1'
2018-02-20 14:28:29 -08:00
a9a9590058
Merge branch 'release-5.1' of github.com:apple/foundationdb into release-5.1
2018-02-16 19:44:36 -08:00
54fc81b260
Improved backup error reporting in backup status. The most recent error for each error type is reported along with how long ago the error occurred, and errors are divided into two categories based on whether or not they occurred since the most recent backup progress.
2018-02-16 19:38:31 -08:00
dc93759e15
suppressed trace events that are spammy
2018-02-16 16:01:19 -08:00
8c53483838
fix: log ranges were not being cleared correctly
2018-02-16 10:27:10 -08:00
0792d5e3dd
Fix: last restorable version for a backup tag name (a separate value from the latest restorable version for a configured backup) was not being updated.
...
Fix: backup blob speed was sometimes an error because the JSON $sum merge operator did not support mixed numeric types.
Fix: JSON merge operator handling was squashing errors in some cases, which was generally obscuring the backup speed metric issue.
Cleaned up some of the JSON object merging logic.
Improved error messages in JSON merge operators. Added JSON merge operator tests for mixed numeric math and improved readability of test output.
2018-02-06 13:44:04 -08:00
4dec5423f7
Optimization in backup snapshot dispatching. If the next dispatch version is not ahead of the recently known current version then do not set a scheduled time for new range tasks in order to avoid the overhead and delay of task timeout handling. Adjusted task timeout knobs to avoid large transaction warnings. Removed backtrace() from LargeTransaction trace event. Tweaked suppression in backup trace events.
2018-01-24 17:40:02 -08:00
95d4e5520b
Added TraceEvent.
2018-01-24 12:44:37 -08:00
aebbe1dcfd
Changed core_versionspersecond knob to int64_t type to avoid integer overflow. Cleaned up backup TraceEvent suppression. Added backtrace to LargeTransaction TraceEvent to make it easier to find the source of large commits in applications using NativeAPI directly.
2018-01-24 11:59:37 -08:00
40d38880fe
Changed version-based folder naming scheme to something simpler, a fixed width 0-padded 19 digit number (the longest a Version can be) with /'s inserted to limit the size of each folder level. Comparisons using these folder names ignore the /'s so any future change to the splitting scheme would still be compatible with the current listing/reading logic.
2018-01-23 15:02:15 -08:00
0ca8b612c7
fix: do not set onDone until all addTaskFutures have had a chance to join that future
2018-01-19 16:29:42 -08:00
9d5f7bd5ab
Aborting an old incompatible backup did not work if the old backup was never actually initialized, started, or was already aborted.
2018-01-19 15:08:03 -08:00
6e96d3c30c
Bug fix, backup snapshots could take unexpectedly long if the desired snapshot interval is less than the configured snapshotDispatch interval.
2018-01-19 12:14:04 -08:00
da02099c4c
Merge branch 'release-5.1' of github.com:apple/foundationdb into release-5.1
2018-01-19 11:02:48 -08:00
fc16bb94ab
Discontinuing a backup that is already restorable now stops immediately and aborts (via validation key) any tasks scheduled to run later.
2018-01-19 11:02:43 -08:00
570f72ba40
fix: nextDispatchVersion was being set too large if the snapshot interval was small
2018-01-19 10:53:58 -08:00
d7f8fe218a
Bug fix, resolving versions to timestamps for use in backup descriptions did not work on a locked database. Added some trace events to AtomicRestore to show progress.
2018-01-17 12:03:19 -08:00
93b34a945f
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 04:09:43 -08:00
f955547796
Bug fixes. Local var int i being declared over a state variable, and iterators being initialized incorrectly.
2018-01-16 11:41:49 -08:00
897ff6f676
Added new knob for how many tasks to add per transaction in backup dispatch, instead of using the value for restore which has much lower overhead per task.
2018-01-16 10:45:21 -08:00
02d72ca4b8
Added yields to CPU-heavy operations in FileBackup's snapshot range dispatcher.
2018-01-16 10:31:44 -08:00
022df3b91b
backup and restore sometimes took too long in simulation
2018-01-09 17:26:42 -08:00
cbeff0f789
Merge branch 'release-5.1' of github.com:apple/foundationdb into release-5.1
2018-01-05 14:13:27 -08:00
2763713cbc
Bug fix, backup snapshot dispatch was calculating that all shards must be done immediately.
2018-01-05 14:12:00 -08:00
0f20068e82
Renamed all TaskBucket backup tasks to more appropriate names. Created the ability to make task aliases and used this to direct old task names to a task definition which will abort backups created before version 5.1.
2018-01-04 22:53:31 -08:00
d43e80cf48
Bug fix, atomicRestore would fail to get a commit version after a commit_unknown_result where the transaction actually was committed. This would cause the restore to target version -1 so it would use one of the first available restorable versions in the backup instead of the version at which the database was locked.
2018-01-03 16:24:02 -08:00
371dee70e6
Improved backup folder structure to be shallower but spread files more uniformly and make each folder's entries lexically sort into version order regardless of numeric length. Improved backup container test to use a random version multiplier on the file versions created in order to test a wide range of versioned folder paths.
2018-01-02 23:22:35 -08:00
f324afc13f
Bug fix in blob store listing when it requires multiple serial requests Added more trace events to FileBackup and BlobStoreEndpoint with suppression and added suppression to existing trace events.
2017-12-22 17:08:25 -08:00
aa8b4c52d5
Removed backup URL from trace events.
2017-12-21 18:22:14 -08:00
69f7409c37
fix: latestRestorable was incorrect
2017-12-21 17:09:21 -08:00
5ed080721d
fix: atomic restore must wait for the restorable version is greater than the lock version
...
fix: latestRestorableVersion calculation was wrong
2017-12-21 15:45:10 -08:00
38cff7d4a5
every transaction which clears applyMutation keys does so on the first proxy
2017-12-20 15:41:47 -08:00
07efdc70c8
more fixes for windows compile
2017-12-20 14:39:23 -08:00
c51de3bb88
fixed windows compile issues
2017-12-20 13:48:31 -08:00
0ab0cf51a3
fix: snapshotDispatch signaled completion after the first snapshot finished
2017-12-20 12:07:35 -08:00
b77276d2f0
First snapshot of a backup should go as fast as possible instead of using the configured snapshot interval.
2017-12-20 01:07:03 -08:00
7caa012fbf
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 00:49:08 -08:00
d87aa521e9
Merge branch 'backup-container-refactor' into continuous-backup
2017-12-19 23:39:00 -08:00
e0d9cea008
Merge branch 'master' into continuous-backup
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
# fdbrpc/BlobStore.actor.cpp
2017-12-19 23:02:14 -08:00
2cd1ff6aae
Bug fix, in restore dispatch the apply lag was being retrieved before updating the apply end version which would make it look like mutations were finished applying early.
2017-12-19 18:11:40 -08:00
61a043ebfa
Added tr->reset() to prevent initial transaction loop attempts from having a higher chance of expiring.
2017-12-19 17:33:45 -08:00
Jingyu Zhou