apple/foundationdb
1
0
Fork 0
mirror of https://github.com/apple/foundationdb.git synced 2025-05-15 18:32:18 +08:00
Code Issues Packages Projects Releases Wiki Activity
8,132 Commits 35 Branches 318 Tags
Commit Graph

101 Commits

Author SHA1 Message Date
Steve Atherton
318b862aa3 Apply clang-format to backup changes. 2021-03-08 01:49:29 -08:00
Steve Atherton
a397d8625f AsyncFileNonDurable now has a flag for whether or not to emulate AIO mode, so the previous behavior limits are preserved when this flag is set. AsyncFileDurable writes which are selected to be durable are again done in a single write, and the logic to handle non-aligned writes for non-AIO mode is more clear and commented. In simulation, backup files on local filesystems are written with random buffer sizes. Backup container unit test now uses more random file sizes and limits memory consumption for all files being written in parallel. 2021-03-07 22:21:10 -08:00
Steve Atherton
1d00c41385 Bug fixes: Avoid empty writes, avoid 0 length truncation, and make AsyncFileNonDurable support writes at arbitrary offsets and lengths. Increase randomness of IBackupFile appends in backup container unit test. 2021-03-05 17:45:36 -08:00
Steve Atherton
1025a0da37 File-based backup now uses async rename to prevent blocking the network thread in the event of slow filesystem metadata operations. Backup now opens its write-only files without a block cache so that writes to disk will be larger (1MB by default instead of 4k). 2021-03-03 21:45:33 -08:00
Steve Atherton
a860778b51 Add write buffering to BackupContainerLocalDirectory::BackupFile to greatly reduce the number of IAsyncFile::write() calls made when writing backup data. The buffer size is controlled by a knob. 2021-01-09 07:57:48 -08:00
Andrew Noyes
6446b4c082 WIP 2020-07-09 22:02:43 +00:00
Steve Atherton
3d72c2a661 BackupContainerFilesystem no longer unnecessarily depends on abspath() to find the last part of a path string, since it shouldn't touch the local filesystem in the remote case. 2020-02-18 16:35:00 -08:00
A.J. Beamon
5f55f3f613 Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used. 2019-05-10 14:01:52 -07:00
Austin Seipp
b5cbffc1b8 fdbclient: fix some print/scan format warnings 
Signed-off-by: Austin Seipp <aseipp@pobox.com>
 2019-05-06 13:35:29 -07:00
mpilman
1c16f87a4e Remove trace-calls to printable (in non-workloads) 2019-04-05 13:12:19 -07:00
Stephen Atherton
cabe7ca844 Stopped using %z to parse timezone offset with strptime() because it only seems to work as expected on MacOS. Updated time input/output unit tests so that they don't assume what the local timezone is. 2019-03-21 19:38:07 -07:00
Stephen Atherton
d5e50e6963 Rewrote BackupAgentBase::parseTime() to use std::get_time() so it compiles on all supported platforms. Added unit test for parseTime() and formatTime(). 2019-03-20 01:18:37 -07:00
Stephen Atherton
c6edcc7f06 Added schema version string to backup JSON status docs. Bug fix in backup status JSON, the document was being created outside the transaction retry loop so retries would combine partial element sets across all tries into the result. 2019-03-14 02:10:14 -07:00
Steve Atherton
8aab719c22
Merge branch 'master' into feature-backup-json 2019-03-12 18:23:16 -07:00
Stephen Atherton
bc0b2aa040 Merge branch 'release-6.0' of https://github.com/apple/foundationdb 
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbclient/BlobStore.actor.cpp
 2019-03-12 04:49:12 -07:00
Stephen Atherton
023bbb566f Renamed backup state enums for clarity, added backup state names. Changed Epochs to EpochSeconds in backup JSON along with some other renaming/moving of fields, and added information about snapshot dispatch. Changed timestamp format for input/output in all backup/restore contexts to be a fully qualified time with timezone offset. Added information about the last snapshot dispatch to backup config and status (not yet populated). 2019-03-10 16:00:01 -07:00
Stephen Atherton
06c11a316d Normalized timestamp to text format across backup and restore tooling. Added epochs field to JSON objects describing versions and timestamps in backup status and describe output, renamed some fields for clarity. 2019-03-06 22:34:25 -08:00
Stephen Atherton
ca8bbad657 Added --json option to fdbbackup describe. Also added expired percentage indicator to snapshot details. 2019-03-06 14:14:06 -08:00
Stephen Atherton
87d0c5bae0 Bug/usability fix: The output URLs of fdbbackup list were meant to be directly usable for backup management operations but they were missing the bucket URL parameter. 2019-03-05 21:14:21 -08:00
Stephen Atherton
d3377722d5 Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup. 2019-03-05 04:00:11 -08:00
Stephen Atherton
7d287c6999 Merge branch 'release-6.0' 
# Conflicts:
#	fdbclient/FileBackupAgent.actor.cpp
 2019-02-28 14:01:00 -08:00
Stephen Atherton
887856b6b0 Bug fix in AsyncFileReadAhead where a file size that is an integer multiple of the read chunk size will cause a crash when reading the file's final block. BackupContainerLocalDirectory now uses AsyncFileReadAhead in simulation to get simulation coverage of that class, and FileBackup will generate file sizes which expose the bug. 2019-02-28 00:22:38 -08:00
mpilman
479a4d7c22 Minor fixes in fdbclient for intellisense 2019-02-19 15:16:59 -08:00
Andrew Noyes
067a445e06 Replace unused _ variables with wait(success(...)) 2019-02-12 17:30:30 -08:00
Alex Miller
0750dc0418 Change store from (Future, T&) to (T&, Future). 
LHS = RHS, and the name of what's being modified is easier to find.
 2019-02-04 18:04:22 -08:00
Evan Tschannen
684a22a52b Merge branch 'release-6.0' 
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbclient/BackupContainer.actor.cpp
#	fdbclient/HTTP.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/BackupCorrectness.actor.cpp
#	versions.target
 2019-01-09 16:14:46 -08:00
Stephen Atherton
bb2de3479b Updated unit test to new backup container behavior. 2019-01-08 16:29:00 -08:00
Stephen Atherton
2951369006 When starting a restore, if range files are missing their names will be logged before an error is thrown. The error thrown is now restore_missing_data instead of restore_corrupted_data. 2019-01-08 16:28:40 -08:00
Stephen Atherton
928876be81 Backup dumpFileList() can now take a version range. 2018-12-21 22:42:29 -08:00
Stephen Atherton
f64d5321e9 Backup describe, expire, and delete will now clearly indicate when there is no backup at the given URL. 2018-12-20 18:05:23 -08:00
Stephen Atherton
354abebf64 Added progress reporting to backup expiration. Simplified backup delete progress. 2018-12-20 00:23:26 -08:00
Stephen Atherton
fcb34a8768 In backup describe, when not using the original cluster to resolve versions to date/time strings the versions will be converted to approximate day deltas from maxLogEndVersion (if available) using core_versionspersecond. 2018-12-19 16:53:39 -08:00
Stephen Atherton
00568f4011 Error code was misleading, added comment. 2018-12-19 13:14:48 -08:00
Stephen Atherton
fd4a62fbfd Backup expire will fail earlier if force option is needed but not specified. 2018-12-19 10:36:25 -08:00
Stephen Atherton
172c3f2021 Bug fix in backup describe, do not update log begin/end metadata in backup if describe was given a start version override as it can result in incorrect metadata. 2018-12-19 10:35:06 -08:00
Stephen Atherton
afa243bc97 Added fdbbackup expire options to calculate approximate version boundaries based on a number of days prior to the latest log file found in the backup container. This enables expiration operations based on time (with reasonable precision) without accessing the source cluster. 2018-12-18 18:55:44 -08:00
Stephen Atherton
69d847dbbd Backup describe can now analyze a backup assuming log data starts at a given version. This is used by expire both as an optimization and because doing so enables expire to repair a bad metadata state that can result from a cancelled or failed expire operation from fdbbackup <= 6.0.17. ListLogFiles() and ListRangeFiles() no longer sort results as in most cases the sort is not required and the result set can be very large. Describe will now update minLogBegin metadata to the beginning of the log range known to be present when possible. Several serial log and range list pairings are now done in parallel. 2018-12-18 04:33:37 -08:00
Stephen Atherton
9ef9041fba Bug fix, metadata read futures were moved to a vector but then not waited on. 2018-12-17 13:13:35 -08:00
Stephen Atherton
dac1827d23 Backup describe's "deep scan" mode should only ignore log begin/end versions, not expire and unreliable end versions. 2018-12-16 00:41:38 -08:00
Stephen Atherton
5951e9d577 Added backup URL and exceptions to trace events where applicable. 2018-12-16 00:33:30 -08:00
Stephen Atherton
223b19f5ba Rewrote backup metadata scheme to fix several design flaws involving cancelled or concurrent operations. Most importantly, before an expire deletes any data it marks the end of the range being deleted as 'unreliable' so further reads of the backup will treat versions before that point as having data holes. 2018-12-16 00:18:13 -08:00
Evan Tschannen
1f3b6e8bdf Merge branch 'release-6.0' 
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/BackupContainer.actor.cpp
#	fdbclient/BlobStore.actor.cpp
#	versions.target
 2018-11-27 14:41:46 -08:00
A.J. Beamon
975711c389 Merge branch 'release-6.0' of github.com:apple/foundationdb 
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
 2018-11-27 09:50:39 -08:00
Stephen Atherton
3d68d6b994 Bug fix, clarified a comment. 2018-11-24 18:41:39 -08:00
Stephen Atherton
0610a19e4d Rewrote backup container unit test to use randomness to cover a wider variety of data patterns and edge cases. 2018-11-24 17:24:54 -08:00
Stephen Atherton
aa648daabf Compile fix for linux, can't make a move iterator from a const container reference. 2018-11-23 12:49:10 -08:00
Stephen Atherton
ec9410492d Changed backup folder scheme to use a much smaller number of unique folders for kv range files, which will speed up list and expire operations. The old scheme is still readable but will no longer be written except in simulation in order to test backward compatibility. 2018-11-23 05:23:56 -08:00
Stephen Atherton
1f2223fcf5 Bug fix in backup expiration. After the range file scan, it was being asserted that the range files found have a version < expiration version but this isn't guaranteed because the expiration version is likely shifted backward a bit after the file scan based on the log files found. 2018-11-15 02:15:25 -08:00
Evan Tschannen
e45952bc53 Merge branch 'release-6.0' 
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/BackupContainer.actor.cpp
#	fdbclient/BlobStore.actor.cpp
#	fdbclient/HTTP.actor.cpp
#	tests/BlobStore.txt
#	versions.target
 2018-11-13 16:06:39 -08:00
Stephen Atherton
3125b807b3 Added documentation of the 'bucket' URL parameter for blobstore:// backup URLs. 2018-11-13 06:23:58 -08:00