5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
b5cbffc1b8
fdbclient: fix some print/scan format warnings
...
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-06 13:35:29 -07:00
1c16f87a4e
Remove trace-calls to printable (in non-workloads)
2019-04-05 13:12:19 -07:00
cabe7ca844
Stopped using %z to parse timezone offset with strptime() because it only seems to work as expected on MacOS. Updated time input/output unit tests so that they don't assume what the local timezone is.
2019-03-21 19:38:07 -07:00
d5e50e6963
Rewrote BackupAgentBase::parseTime() to use std::get_time() so it compiles on all supported platforms. Added unit test for parseTime() and formatTime().
2019-03-20 01:18:37 -07:00
c6edcc7f06
Added schema version string to backup JSON status docs. Bug fix in backup status JSON, the document was being created outside the transaction retry loop so retries would combine partial element sets across all tries into the result.
2019-03-14 02:10:14 -07:00
8aab719c22
Merge branch 'master' into feature-backup-json
2019-03-12 18:23:16 -07:00
bc0b2aa040
Merge branch 'release-6.0' of https://github.com/apple/foundationdb
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BlobStore.actor.cpp
2019-03-12 04:49:12 -07:00
023bbb566f
Renamed backup state enums for clarity, added backup state names. Changed Epochs to EpochSeconds in backup JSON along with some other renaming/moving of fields, and added information about snapshot dispatch. Changed timestamp format for input/output in all backup/restore contexts to be a fully qualified time with timezone offset. Added information about the last snapshot dispatch to backup config and status (not yet populated).
2019-03-10 16:00:01 -07:00
06c11a316d
Normalized timestamp to text format across backup and restore tooling. Added epochs field to JSON objects describing versions and timestamps in backup status and describe output, renamed some fields for clarity.
2019-03-06 22:34:25 -08:00
ca8bbad657
Added --json option to fdbbackup describe. Also added expired percentage indicator to snapshot details.
2019-03-06 14:14:06 -08:00
87d0c5bae0
Bug/usability fix: The output URLs of fdbbackup list were meant to be directly usable for backup management operations but they were missing the bucket URL parameter.
2019-03-05 21:14:21 -08:00
d3377722d5
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 04:00:11 -08:00
7d287c6999
Merge branch 'release-6.0'
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
2019-02-28 14:01:00 -08:00
887856b6b0
Bug fix in AsyncFileReadAhead where a file size that is an integer multiple of the read chunk size will cause a crash when reading the file's final block. BackupContainerLocalDirectory now uses AsyncFileReadAhead in simulation to get simulation coverage of that class, and FileBackup will generate file sizes which expose the bug.
2019-02-28 00:22:38 -08:00
479a4d7c22
Minor fixes in fdbclient for intellisense
2019-02-19 15:16:59 -08:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
0750dc0418
Change store from (Future, T&) to (T&, Future).
...
LHS = RHS, and the name of what's being modified is easier to find.
2019-02-04 18:04:22 -08:00
684a22a52b
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BackupContainer.actor.cpp
# fdbclient/HTTP.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/BackupCorrectness.actor.cpp
# versions.target
2019-01-09 16:14:46 -08:00
bb2de3479b
Updated unit test to new backup container behavior.
2019-01-08 16:29:00 -08:00
2951369006
When starting a restore, if range files are missing their names will be logged before an error is thrown. The error thrown is now restore_missing_data instead of restore_corrupted_data.
2019-01-08 16:28:40 -08:00
928876be81
Backup dumpFileList() can now take a version range.
2018-12-21 22:42:29 -08:00
f64d5321e9
Backup describe, expire, and delete will now clearly indicate when there is no backup at the given URL.
2018-12-20 18:05:23 -08:00
354abebf64
Added progress reporting to backup expiration. Simplified backup delete progress.
2018-12-20 00:23:26 -08:00
fcb34a8768
In backup describe, when not using the original cluster to resolve versions to date/time strings the versions will be converted to approximate day deltas from maxLogEndVersion (if available) using core_versionspersecond.
2018-12-19 16:53:39 -08:00
00568f4011
Error code was misleading, added comment.
2018-12-19 13:14:48 -08:00
fd4a62fbfd
Backup expire will fail earlier if force option is needed but not specified.
2018-12-19 10:36:25 -08:00
172c3f2021
Bug fix in backup describe, do not update log begin/end metadata in backup if describe was given a start version override as it can result in incorrect metadata.
2018-12-19 10:35:06 -08:00
afa243bc97
Added fdbbackup expire options to calculate approximate version boundaries based on a number of days prior to the latest log file found in the backup container. This enables expiration operations based on time (with reasonable precision) without accessing the source cluster.
2018-12-18 18:55:44 -08:00
69d847dbbd
Backup describe can now analyze a backup assuming log data starts at a given version. This is used by expire both as an optimization and because doing so enables expire to repair a bad metadata state that can result from a cancelled or failed expire operation from fdbbackup <= 6.0.17. ListLogFiles() and ListRangeFiles() no longer sort results as in most cases the sort is not required and the result set can be very large. Describe will now update minLogBegin metadata to the beginning of the log range known to be present when possible. Several serial log and range list pairings are now done in parallel.
2018-12-18 04:33:37 -08:00
9ef9041fba
Bug fix, metadata read futures were moved to a vector but then not waited on.
2018-12-17 13:13:35 -08:00
dac1827d23
Backup describe's "deep scan" mode should only ignore log begin/end versions, not expire and unreliable end versions.
2018-12-16 00:41:38 -08:00
5951e9d577
Added backup URL and exceptions to trace events where applicable.
2018-12-16 00:33:30 -08:00
223b19f5ba
Rewrote backup metadata scheme to fix several design flaws involving cancelled or concurrent operations. Most importantly, before an expire deletes any data it marks the end of the range being deleted as 'unreliable' so further reads of the backup will treat versions before that point as having data holes.
2018-12-16 00:18:13 -08:00
1f3b6e8bdf
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/BackupContainer.actor.cpp
# fdbclient/BlobStore.actor.cpp
# versions.target
2018-11-27 14:41:46 -08:00
975711c389
Merge branch 'release-6.0' of github.com:apple/foundationdb
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2018-11-27 09:50:39 -08:00
3d68d6b994
Bug fix, clarified a comment.
2018-11-24 18:41:39 -08:00
0610a19e4d
Rewrote backup container unit test to use randomness to cover a wider variety of data patterns and edge cases.
2018-11-24 17:24:54 -08:00
aa648daabf
Compile fix for linux, can't make a move iterator from a const container reference.
2018-11-23 12:49:10 -08:00
ec9410492d
Changed backup folder scheme to use a much smaller number of unique folders for kv range files, which will speed up list and expire operations. The old scheme is still readable but will no longer be written except in simulation in order to test backward compatibility.
2018-11-23 05:23:56 -08:00
1f2223fcf5
Bug fix in backup expiration. After the range file scan, it was being asserted that the range files found have a version < expiration version but this isn't guaranteed because the expiration version is likely shifted backward a bit after the file scan based on the log files found.
2018-11-15 02:15:25 -08:00
e45952bc53
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/BackupContainer.actor.cpp
# fdbclient/BlobStore.actor.cpp
# fdbclient/HTTP.actor.cpp
# tests/BlobStore.txt
# versions.target
2018-11-13 16:06:39 -08:00
3125b807b3
Added documentation of the 'bucket' URL parameter for blobstore:// backup URLs.
2018-11-13 06:23:58 -08:00
983bd3346a
BackupContainerBlobStore no longer uses a hardcoded bucket name. BlobStoreEndpoint creating from a URL string now supports having additional parameters in the URL which it does not consume but rather returns to the caller, and BackupContainerBlobStore uses this to accept a "bucket" parameter.
2018-11-13 03:00:59 -08:00
4b5d0b4e2c
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/AsyncFileBlobStore.actor.cpp
# fdbclient/AsyncFileBlobStore.actor.h
# fdbclient/BlobStore.actor.cpp
# fdbclient/BlobStore.h
# fdbclient/HTTP.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/batcher.actor.h
# fdbrpc/fdbrpc.vcxproj
# fdbrpc/sim2.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2018-11-10 13:04:24 -08:00
776b289bfe
Move AsyncFileBlobStore and related files to fdbclient.
2018-10-26 13:49:42 -07:00
268093a96d
Adjust all includes to be relative to the root.
...
Remove the use of relative paths. A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h". Adjust so that every include references such a header with the
latter form.
Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
22f8a4efa9
Normalized all unit test names to begin with "/" if they should be included in random unit testing.
2018-10-05 22:09:58 -07:00
3922e477a5
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/LogSystemDiskQueueAdapter.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
2018-10-03 16:57:18 -07:00
3f86905ea7
fix: restore did not take into account that the end version of a log file does not exist in that file. This resulted in restores done at the same version a snapshot completes to not apply the mutations at that final version.
2018-09-21 11:48:28 -07:00
A.J. Beamon