apple/foundationdb
1
0
Fork 0
mirror of https://github.com/apple/foundationdb.git synced 2025-06-03 03:41:53 +08:00
Code Issues Packages Projects Releases Wiki Activity
1,016 Commits 35 Branches 318 Tags
Commit Graph

151 Commits

Author SHA1 Message Date
Alec Grieser
e1162e9238 Merge remote-tracking branch 'upstream/release-5.1' 2018-02-22 11:16:12 -08:00
Alec Grieser
0bae9880f1 remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py 2018-02-21 10:25:11 -08:00
Alec Grieser
aadc06de99 Merge remote-tracking branch 'upstream/release-5.1' 2018-02-20 14:28:29 -08:00
Alec Grieser
1c1ae7d70e Merge remote-tracking branch 'upstream/release-5.1' into bindings-format 2018-02-19 12:37:06 -08:00
Evan Tschannen
dc93759e15 suppressed trace events that are spammy 2018-02-16 16:01:19 -08:00
A.J. Beamon
814ae16016 Add destination tokens to Net2_LargePacket trace events. Add backtrace when a sent packet is too large. 2018-02-15 14:54:35 -08:00
Balachandar Namasivayam
f320b1b347 Change ConnectionClosed TraceEvent severity from SevError to SevWarnAlways. 2018-02-14 12:25:54 -08:00
Stephen Atherton
69425a303b Improved error handling for cases where blob account credentials are either not found in the provided credentials sources and/or some of the credentials sources provided are not readable or parseable. 2018-02-07 21:50:43 -08:00
Stephen Atherton
f8522248cb Blob credentials files were being opened in read-write mode despite the read-only option being specified because the underlying caching layer opens always opens files for read/write access. For now, disabled caching for this file. 2018-02-07 16:25:16 -08:00
Stephen Atherton
d8879dc3f3 HTTP::doRequest() now reads responses in parallel with sending requests, so if the server responds before receiving all of the the request the client can stop sending the remainder of the request. For PUT requests which upload files, this prevents sending potentially several megabytes of unnecessary bytes if the server responds with an error (such as 429) before the request is completely sent. Updated the backup container unit test to use more parallelism in order to test this new behavior. 2018-02-07 10:38:31 -08:00
Stephen Atherton
0792d5e3dd Fix: last restorable version for a backup tag name (a separate value from the latest restorable version for a configured backup) was not being updated. 
Fix: backup blob speed was sometimes an error because the JSON $sum merge operator did not support mixed numeric types.
Fix: JSON merge operator handling was squashing errors in some cases, which was generally obscuring the backup speed metric issue.
Cleaned up some of the JSON object merging logic.
Improved error messages in JSON merge operators.  Added JSON merge operator tests for mixed numeric math and improved readability of test output.
 2018-02-06 13:44:04 -08:00
Stephen Atherton
2f291d8955 Bug fix in blob backup container deletion. The list/delete loop could end before deleting all of the files, but the index entry would still be deleted. Also preemptively made the same code change in listBucket() - Although it is technically correct as written it is a dangerous style because it is not obvious that the addition of a wait() call in the second 'when' block would create a bug. Consolidated deleteContainer() and deleteBucket() as they differ by only 1 line. 2018-01-29 00:32:41 -08:00
Alec Grieser
51781bb7a8 Merge branch 'release-5.1' into bindings-format 2018-01-26 12:28:29 -08:00
Stephen Atherton
9fd2a8df3d Tweaked a trace event suppression time. 2018-01-24 19:08:24 -08:00
Alec Grieser
57986cfe00 format python files to be roughtly pep8 compliant 2018-01-24 19:06:58 -08:00
A.J. Beamon
19ed388c0e Merge branch 'release-5.0' into release-5.1 
# Conflicts:
#	documentation/sphinx/source/downloads.rst
#	documentation/sphinx/source/release-notes.rst
#	versions.target
 2018-01-24 14:43:41 -08:00
Stephen Atherton
7f18d59dfe Bug fix, the blob request attempt count is now incremented for all errors except response code 429. 2018-01-24 01:15:01 -08:00
Stephen Atherton
a2481343ec Bug fix, HTTP error code 429 was not being considered retryable in blob client (this was previously fixed but apparently reintroduced). 2018-01-24 00:22:11 -08:00
Stephen Atherton
66de9d392b New error code, http_auth_failed, which is used when blob authentication fails instead of the previous generic http_request_failed. 2018-01-22 14:58:56 -08:00
Stephen Atherton
307e04c0ad Updated backup container unit test to match new safer behavior of expireData(). Rewrote BackupContainerLocalDirectory::deleteContainer() to actually delete the whole directory but only if it appears to be a backup with either log or snapshot data. 2018-01-18 00:36:28 -08:00
Stephen Atherton
93b34a945f Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup. 2018-01-17 04:09:43 -08:00
Alvin Moore
2e6ce03224 Merge pull request #232 from cie/build-dont-compile-hpp 
Filter out .hpp files from *_BUILD_SOURCES (like we do with .h files)…
 2018-01-12 14:09:25 -08:00
Evan Tschannen
02bd83ff76 changed incompatibleDataRead to an asyncTrigger 2018-01-11 13:35:56 -08:00
A.J. Beamon
80b84c23ac Filter out .hpp files from *_BUILD_SOURCES (like we do with .h files). Add xml2json.hpp to our fdbrpc project. 2018-01-10 13:51:57 -08:00
A.J. Beamon
ce93d98b50 Temporarily remove xml2json.hpp from fdbrpc vcxproj 2018-01-10 10:18:44 -08:00
A.J. Beamon
2f5073d00f Some visual studio project cleanup. 2018-01-10 10:07:18 -08:00
Stephen Atherton
0e7d538c94 Bug fix, in recursive blob folder listings the recent removal of common prefixes from the result stream caused the list marker to not be set correctly when a folder level requires multiple requests due to folder size. 2018-01-06 20:58:48 -08:00
Stephen Atherton
96cb06cbc7 Bug fixes. Fdbbackup delete was broken. Blobstore backup container deletion would do too much listing before deletions began due to list operations queueing up ahead of and starving the delete operations. Created new knob and blob endpoint limit for concurrent list operations to fix this. Increased blob request timeout default because some requests were taking longer. Crash fixes in blobstore doRequest() which wasn't checking that response object is valid before using it in error conditions. Filesystem-like backup container class (covering blobstore and local dirs) now ignores unrecognized filenames for describe() and expire() operations. 2018-01-05 23:06:39 -08:00
Stephen Atherton
78430425e8 Blob bucket listings will now use parallel recursive requests on CommonPrefixes, up to a max depth, if a delimiter is provided. 2018-01-02 23:17:52 -08:00
Stephen Atherton
07fde9dfb4 Bug fix, error code 429 was not being treated as retryable in the recent refactor. 2018-01-02 23:15:25 -08:00
Stephen Atherton
f324afc13f Bug fix in blob store listing when it requires multiple serial requests Added more trace events to FileBackup and BlobStoreEndpoint with suppression and added suppression to existing trace events. 2017-12-22 17:08:25 -08:00
Stephen Atherton
f2524ffd33 AsyncFileBlobStoreWrite was prohibiting the writing of 0-byte files. Improved HTTP verbose logging to stdout. Added writing a 0-byte file to BackupContainer unit test. Added backup log and snapshot sizes to backup description. 2017-12-21 21:15:26 -08:00
Stephen Atherton
e0ef5a9a20 Whitespace normalization. 2017-12-21 12:07:29 -08:00
Stephen Atherton
e3aee45a74 Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings. 2017-12-21 01:58:15 -08:00
Stephen Atherton
e0d9cea008 Merge branch 'master' into continuous-backup 
# Conflicts:
#	fdbclient/FileBackupAgent.actor.cpp
#	fdbrpc/BlobStore.actor.cpp
 2017-12-19 23:02:14 -08:00
Alex Miller
9a0df6d76d Deallocate aligned_alloc with aligned_free. 
This probably fixes a windows-only crash, as only windows cares about this distinction.
 2017-12-14 15:12:05 -08:00
Stephen Atherton
b6cfe010a1 Bug fix in URL encoding of delimiter. 2017-12-12 17:31:19 -08:00
Stephen Atherton
872edd7540 Merge branch 'release-5.0' 
# Conflicts:
#	fdbclient/FileBackupAgent.actor.cpp
 2017-12-06 16:27:04 -08:00
Stephen Atherton
41f80bf7ed Renamed an error, changed blob request failure to Warn severity. 2017-12-06 15:58:54 -08:00
Stephen Atherton
4bc7d0b86a Updated error names and severities. 2017-12-06 15:42:44 -08:00
Stephen Atherton
abb2dd1ebc Merge pull request #214 from cie/alexmiller/fallocate 
Use fallocate to zero ranges instead of writing zeroes
 2017-12-06 13:47:40 -08:00
Alex Miller
064670a95b Maintain a reference to the IAsyncFile in zeroRange. 
And also add some notes about the reference semantics to the IAsyncFile header
for future readers.
 2017-12-06 13:41:21 -08:00
Balachandar Namasivayam
1f949240f5 Make fdbbackup s3 compatible. 
s3 sends response in XML.  FDB backup expects json response. Added a new libraray xml2json to convert xml to json.
 2017-12-05 17:13:15 -08:00
Stephen Atherton
86ae6c09c7 Bug fixes, take(1) is incorrect usage of FlowLock. 2017-12-04 10:20:50 -08:00
Evan Tschannen
482ac38ca6 added knobs so that the client failure monitoring update rate and the server failure monitoring update rate are separate knobs 2017-12-01 13:04:32 -08:00
Alex Miller
7bab3a4ece AsyncFileKAIO will prefer using fallocate's ZERO_RANGE for AsyncFile::zero(). 
For situations in which we have support for FALLOC_FL_ZERO_RANGE, it's much
faster to use fallocate than manually overwrite the file with zero bytes.  Note
that this support depends on having a kernel from late 2014 or newer, and being
on ext4 or xfs.  If these conditions aren't met, we'll fall back to writing
zeros in 1MB chunks as normal.
 2017-11-30 17:57:55 -08:00
Alex Miller
196258080b Refactor zeroing a chunk of a file from DiskQueue into IAsyncFile. 
If we're going to do the work to provide more optimized ways to zero files,
then I'd feel better with this being in a more common place, so that any other
zero-ers are likely to reuse it.  It also makes testing easier/more obvious.

Also, because it's needed for correctness, fix the aligned_alloc for OSX, which
wasn't aligned, and use an actually aligned allocation function.
 2017-11-30 17:57:55 -08:00
Alex Miller
c7a120c59d Rename IAsyncFile::incrementalDelete -> IAsyncFileSystem::incrementalDeleteFile. 
`deleteFile` existed in IAsyncFileSystem, so an incremental delete function
seems to belong more as a virtual method on IAsyncFileSystem than a static
method on IAsyncFile, and the naming should match.

As long as we're here, change IAsyncFile to declare a virtual destructor, so
that it has good and proper C++ behavior.  I presume this is what was vaguely
intended by the default constructor definition that previously existed?
 2017-11-30 17:19:10 -08:00
Stephen Atherton
1e643239f9 Improvement in blob connnection reuse, oldest connnections in pool are now used first. 2017-11-30 12:57:29 -08:00
Stephen Atherton
1b1c8e985a Merge branch 'master' into backup-container-refactor 
# Conflicts:
#	fdbclient/FileBackupAgent.actor.cpp
 2017-11-25 19:54:51 -08:00