f6559bbb11
Added help and documentation for fdbbackup modify. Add BackupUID and more parseable BackupURL to fdbbackup status output.
2019-03-05 05:13:48 -08:00
d3377722d5
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 04:00:11 -08:00
5231ee6da0
make sure the version of a cluster is larger than the restore version, so future version stamps are larger than restored version stamps
2019-03-03 21:34:21 -08:00
2168b14834
Merge branch 'master' into feature-metadata-version
2019-02-28 17:45:55 -08:00
3da85f3acd
implemented the \xff/metadataVersion key, which can be used by layers to help them cheaply cache metadata and know when their cache is invalid
2019-02-28 17:45:00 -08:00
7d287c6999
Merge branch 'release-6.0'
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
2019-02-28 14:01:00 -08:00
887856b6b0
Bug fix in AsyncFileReadAhead where a file size that is an integer multiple of the read chunk size will cause a crash when reading the file's final block. BackupContainerLocalDirectory now uses AsyncFileReadAhead in simulation to get simulation coverage of that class, and FileBackup will generate file sizes which expose the bug.
2019-02-28 00:22:38 -08:00
7eba50b086
Add support for restoring multiple ranges.
2019-02-25 18:00:28 -08:00
4be90c91d6
Moved THIS and THIS_ADDR to actorcompiler.h
2019-02-19 15:16:59 -08:00
f3cd9f15ca
Update fdbclient/FileBackupAgent.actor.cpp
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-02-19 15:16:59 -08:00
82cfb94f4a
Update fdbclient/FileBackupAgent.actor.cpp
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-02-19 15:16:59 -08:00
f79a9594c1
Several bugfixes to make fdb build on non-ide
2019-02-19 15:16:59 -08:00
7fa7db3251
Minor bug fix - fdbclient now compiling
2019-02-19 15:16:59 -08:00
c1592b4c3a
Several minor fixes within fdbclient
2019-02-19 15:16:59 -08:00
70e089e791
Define THIS_ADDR for use with intellisense
...
this is undefined within ACTORs if compiled with
intellisense. We use a macro here to make sure just
replaces this with 0
2019-02-19 15:16:59 -08:00
78dd80ea8a
Proper fwd decl in BackupAgent
...
Also BackupAgent.h -> BackupAgent.actor.h
2019-02-19 15:16:59 -08:00
3cb2391b58
use proper fwd declarations in ManagementAPI
...
Also ManagementAPI.h -> ManagementAPI.actor.h
2019-02-19 15:16:59 -08:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
0750dc0418
Change store from (Future, T&) to (T&, Future).
...
LHS = RHS, and the name of what's being modified is easier to find.
2019-02-04 18:04:22 -08:00
684a22a52b
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BackupContainer.actor.cpp
# fdbclient/HTTP.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/BackupCorrectness.actor.cpp
# versions.target
2019-01-09 16:14:46 -08:00
604ad062d5
Updated backup correctness test to new behavior. WaitBackup() can now return the UID and BackupContainer atomically with the status code for a backup tag.
2019-01-08 18:12:15 -08:00
d2d68aa171
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# versions.target
2018-12-03 18:26:52 -08:00
e974ee367d
Bug fix, backups in progress during upgrade to 6.0.16 or later do not have have range file count initialized so reading it must default to 0.
2018-12-02 22:50:26 -08:00
1f3b6e8bdf
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/BackupContainer.actor.cpp
# fdbclient/BlobStore.actor.cpp
# versions.target
2018-11-27 14:41:46 -08:00
ec9410492d
Changed backup folder scheme to use a much smaller number of unique folders for kv range files, which will speed up list and expire operations. The old scheme is still readable but will no longer be written except in simulation in order to test backward compatibility.
2018-11-23 05:23:56 -08:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
0217aed74c
Merge branch 'release-6.0'
...
# Conflicts:
# bindings/go/README.md
# documentation/sphinx/source/release-notes.rst
# fdbserver/MasterProxyServer.actor.cpp
# versions.target
2018-10-15 18:38:51 -07:00
770b4b858b
Minor (and asymptomatic) bug fixes where ints or enumerations are being passed as the bool snapshot argument instead of as the default value which was intended.
2018-10-12 15:03:21 -07:00
3922e477a5
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/LogSystemDiskQueueAdapter.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
2018-10-03 16:57:18 -07:00
3f86905ea7
fix: restore did not take into account that the end version of a log file does not exist in that file. This resulted in restores done at the same version a snapshot completes to not apply the mutations at that final version.
2018-09-21 11:48:28 -07:00
e1fafbf259
Add some missing actorcompiler.h includes
2018-08-22 09:40:45 -07:00
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
1c29275672
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-01 14:30:57 -07:00
99c9958db7
Some more trace event normalization
2018-06-08 13:57:00 -07:00
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
e82985aea2
fix: continue setting beginVersion so that versions between 5.2.0 and 5.2.2 do not crash when decoding tasks created by 5.2.3
2018-06-06 13:34:22 -07:00
4120062bb9
fix: backup initialized its begin version at 1 instead of the read version of the starting transaction
...
fix: erasing log ranges did not properly divide up work between transactions to prevent making transactions which were too large
2018-06-06 13:05:53 -07:00
c37b48308e
Fix the way to get tagName
2018-05-03 11:11:44 -07:00
e5320fa237
Merge pull request #292 from ajbeamon/printable-encode-backup-beginfile
...
Encode BeginFile TraceEvent detail with printable
2018-05-01 15:53:48 -07:00
f848c68b73
TraceEvents in FileBackupAgent that include BeginFile should be encoding them with printable because of the possible inclusion of the null byte. This causes the detail to terminate prematurely.
2018-05-01 15:35:48 -07:00
52cc592d5e
Abort 5.1 backup for 5.2 upgrade
2018-05-01 14:49:14 -07:00
8d7b9fd506
Merge branch 'release-5.2' of github.com:apple/foundationdb into optimize-backup-priorities
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
2018-03-23 12:10:39 -07:00
ede5cab192
Merge pull request #89 from yichic/share-log-mutations-5.2
...
Share log mutations 5.2
2018-03-19 12:01:26 -07:00
ec02e54f64
Refactor EraseLogData()
2018-03-19 11:56:01 -07:00
1f2602d2b3
Fix all review comments
2018-03-19 11:33:33 -07:00
d6559b144f
Share log mutations between backups and DRs which have the same backup range
2018-03-19 11:32:50 -07:00
c5bb3421a7
Optimized backup task priorities so that during the initial snapshot range tasks are prioritized and then afterwards log tasks are prioritized.
2018-03-13 21:49:25 -07:00
94e90454da
Fix line endings
2018-03-05 12:16:37 -08:00
066a9b7e55
Declare each state variable on its own line...
2018-03-02 16:30:13 -08:00
34715ec1fa
Improve fdbbackup status
...
Additional details added are
log bytes written
range bytes written
last snapshot version and its corresponding timestamp
last log version and its corresponding timestamp
If backup is supposed to stop at the next snapshot completion (stopWhenDone)
version/timestamp of snapshot start and expected end
Backup error reporting format changed to
Recent Errors (since latest restore point TIME ago)
<TIME> ago: <error description> on <task name>
Older Errors
<TIME> ago: <error description> on <task name>
TIME format:
<= 59 seconds ago
<= 59.99 minutes ago
<= 23.99 hours ago
N.NN days ago
2018-02-28 18:22:45 -08:00
e1162e9238
Merge remote-tracking branch 'upstream/release-5.1'
2018-02-22 11:16:12 -08:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
aadc06de99
Merge remote-tracking branch 'upstream/release-5.1'
2018-02-20 14:28:29 -08:00
a9a9590058
Merge branch 'release-5.1' of github.com:apple/foundationdb into release-5.1
2018-02-16 19:44:36 -08:00
54fc81b260
Improved backup error reporting in backup status. The most recent error for each error type is reported along with how long ago the error occurred, and errors are divided into two categories based on whether or not they occurred since the most recent backup progress.
2018-02-16 19:38:31 -08:00
dc93759e15
suppressed trace events that are spammy
2018-02-16 16:01:19 -08:00
8c53483838
fix: log ranges were not being cleared correctly
2018-02-16 10:27:10 -08:00
0792d5e3dd
Fix: last restorable version for a backup tag name (a separate value from the latest restorable version for a configured backup) was not being updated.
...
Fix: backup blob speed was sometimes an error because the JSON $sum merge operator did not support mixed numeric types.
Fix: JSON merge operator handling was squashing errors in some cases, which was generally obscuring the backup speed metric issue.
Cleaned up some of the JSON object merging logic.
Improved error messages in JSON merge operators. Added JSON merge operator tests for mixed numeric math and improved readability of test output.
2018-02-06 13:44:04 -08:00
4dec5423f7
Optimization in backup snapshot dispatching. If the next dispatch version is not ahead of the recently known current version then do not set a scheduled time for new range tasks in order to avoid the overhead and delay of task timeout handling. Adjusted task timeout knobs to avoid large transaction warnings. Removed backtrace() from LargeTransaction trace event. Tweaked suppression in backup trace events.
2018-01-24 17:40:02 -08:00
95d4e5520b
Added TraceEvent.
2018-01-24 12:44:37 -08:00
aebbe1dcfd
Changed core_versionspersecond knob to int64_t type to avoid integer overflow. Cleaned up backup TraceEvent suppression. Added backtrace to LargeTransaction TraceEvent to make it easier to find the source of large commits in applications using NativeAPI directly.
2018-01-24 11:59:37 -08:00
40d38880fe
Changed version-based folder naming scheme to something simpler, a fixed width 0-padded 19 digit number (the longest a Version can be) with /'s inserted to limit the size of each folder level. Comparisons using these folder names ignore the /'s so any future change to the splitting scheme would still be compatible with the current listing/reading logic.
2018-01-23 15:02:15 -08:00
0ca8b612c7
fix: do not set onDone until all addTaskFutures have had a chance to join that future
2018-01-19 16:29:42 -08:00
9d5f7bd5ab
Aborting an old incompatible backup did not work if the old backup was never actually initialized, started, or was already aborted.
2018-01-19 15:08:03 -08:00
6e96d3c30c
Bug fix, backup snapshots could take unexpectedly long if the desired snapshot interval is less than the configured snapshotDispatch interval.
2018-01-19 12:14:04 -08:00
da02099c4c
Merge branch 'release-5.1' of github.com:apple/foundationdb into release-5.1
2018-01-19 11:02:48 -08:00
fc16bb94ab
Discontinuing a backup that is already restorable now stops immediately and aborts (via validation key) any tasks scheduled to run later.
2018-01-19 11:02:43 -08:00
570f72ba40
fix: nextDispatchVersion was being set too large if the snapshot interval was small
2018-01-19 10:53:58 -08:00
d7f8fe218a
Bug fix, resolving versions to timestamps for use in backup descriptions did not work on a locked database. Added some trace events to AtomicRestore to show progress.
2018-01-17 12:03:19 -08:00
93b34a945f
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 04:09:43 -08:00
f955547796
Bug fixes. Local var int i being declared over a state variable, and iterators being initialized incorrectly.
2018-01-16 11:41:49 -08:00
897ff6f676
Added new knob for how many tasks to add per transaction in backup dispatch, instead of using the value for restore which has much lower overhead per task.
2018-01-16 10:45:21 -08:00
02d72ca4b8
Added yields to CPU-heavy operations in FileBackup's snapshot range dispatcher.
2018-01-16 10:31:44 -08:00
022df3b91b
backup and restore sometimes took too long in simulation
2018-01-09 17:26:42 -08:00
cbeff0f789
Merge branch 'release-5.1' of github.com:apple/foundationdb into release-5.1
2018-01-05 14:13:27 -08:00
2763713cbc
Bug fix, backup snapshot dispatch was calculating that all shards must be done immediately.
2018-01-05 14:12:00 -08:00
0f20068e82
Renamed all TaskBucket backup tasks to more appropriate names. Created the ability to make task aliases and used this to direct old task names to a task definition which will abort backups created before version 5.1.
2018-01-04 22:53:31 -08:00
d43e80cf48
Bug fix, atomicRestore would fail to get a commit version after a commit_unknown_result where the transaction actually was committed. This would cause the restore to target version -1 so it would use one of the first available restorable versions in the backup instead of the version at which the database was locked.
2018-01-03 16:24:02 -08:00
371dee70e6
Improved backup folder structure to be shallower but spread files more uniformly and make each folder's entries lexically sort into version order regardless of numeric length. Improved backup container test to use a random version multiplier on the file versions created in order to test a wide range of versioned folder paths.
2018-01-02 23:22:35 -08:00
f324afc13f
Bug fix in blob store listing when it requires multiple serial requests Added more trace events to FileBackup and BlobStoreEndpoint with suppression and added suppression to existing trace events.
2017-12-22 17:08:25 -08:00
aa8b4c52d5
Removed backup URL from trace events.
2017-12-21 18:22:14 -08:00
69f7409c37
fix: latestRestorable was incorrect
2017-12-21 17:09:21 -08:00
5ed080721d
fix: atomic restore must wait for the restorable version is greater than the lock version
...
fix: latestRestorableVersion calculation was wrong
2017-12-21 15:45:10 -08:00
38cff7d4a5
every transaction which clears applyMutation keys does so on the first proxy
2017-12-20 15:41:47 -08:00
07efdc70c8
more fixes for windows compile
2017-12-20 14:39:23 -08:00
c51de3bb88
fixed windows compile issues
2017-12-20 13:48:31 -08:00
0ab0cf51a3
fix: snapshotDispatch signaled completion after the first snapshot finished
2017-12-20 12:07:35 -08:00
b77276d2f0
First snapshot of a backup should go as fast as possible instead of using the configured snapshot interval.
2017-12-20 01:07:03 -08:00
7caa012fbf
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 00:49:08 -08:00
d87aa521e9
Merge branch 'backup-container-refactor' into continuous-backup
2017-12-19 23:39:00 -08:00
e0d9cea008
Merge branch 'master' into continuous-backup
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
# fdbrpc/BlobStore.actor.cpp
2017-12-19 23:02:14 -08:00
2cd1ff6aae
Bug fix, in restore dispatch the apply lag was being retrieved before updating the apply end version which would make it look like mutations were finished applying early.
2017-12-19 18:11:40 -08:00
61a043ebfa
Added tr->reset() to prevent initial transaction loop attempts from having a higher chance of expiring.
2017-12-19 17:33:45 -08:00
aa5169bd3c
Removed unnecessary trace event.
2017-12-19 15:29:22 -08:00
e28641886d
TraceEvent improvements. Minor bug fix, restore log writing tasks didn't have the log file endVersion but it's only for logging purposes.
2017-12-19 15:27:04 -08:00
a276985baf
Bug fix, if there are range files in a restore which begin at exactly the restore version they will be repeatedly dispatched forever.
2017-12-18 17:48:18 -08:00
005a4a0706
Restore status bug fix, during restore the apply lag would appear as a large negative number until the first restore batch is completed. Test improvement, snapshot dispatch now chooses a random number of tasks to dispatch per commit.
2017-12-18 15:56:57 -08:00
937fa75bec
Bug fix, if target snapshot end version is at or before the begin version then no progress would be made.
2017-12-18 00:13:25 -08:00
d32a770648
Bug fix, backup never went to differential mode once it was restorable which caused waitBackup to only return once the backup was discontinued.
2017-12-17 23:22:18 -08:00
2b92815e8c
Bug fix. The snapshot dispatch add task retry loop was incorrectly deciding that the second and further transaction of an execution was already committed and therefore skipping it, resulting in missing ranges in the snapshot.
2017-12-17 21:01:31 -08:00
Stephen Atherton