apple/foundationdb
1
0
Fork 0
mirror of https://github.com/apple/foundationdb.git synced 2025-06-03 03:41:53 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,412 Commits 35 Branches 318 Tags
Commit Graph

252 Commits

Author SHA1 Message Date
Evan Tschannen
f17f00fdd5 Merge branch 'release-6.2' 
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
 2020-05-10 22:33:38 -07:00
Meng Xu
c237ca9708
Merge pull request #3081 from jzhou77/backup-cmd 
Update fdbbackup documentation on new "-p" option
 2020-05-04 14:17:34 -07:00
Jingyu Zhou
2c5da1c116 Remove "-p" option and use "--partitioned_log_experimental" option 2020-05-04 13:45:54 -07:00
Alex Miller
78d644f673
Merge branch 'master' into bugfixes/cmake-includes 2020-05-04 02:11:46 -07:00
Meng Xu
37a537c2a3 Fix MacOS compilation error 2020-04-30 18:30:15 -07:00
Steve Atherton
02aeab1edd Re-add improperly named options in order to not break compatibility with anything depending on them in a patch release. 2020-04-30 16:23:30 -07:00
Meng Xu
07a9a05683 FastRestore:Agent:Fix restore requests 2020-04-30 16:20:20 -07:00
Steve Atherton
697beb3ebb Fix fdbrestore parameters with one leading hyphen instead of two, which did not match documentation. 2020-04-30 12:13:35 -07:00
Meng Xu
3f510d0653
Merge pull request #3036 from jzhou77/backup-cmd 
Several bug fixes for new backups
 2020-04-29 16:34:59 -07:00
Meng Xu
e09394459a Merge branch 'master' into mengxu/fr-circus 2020-04-28 22:08:29 -07:00
Meng Xu
8742cc0ab6 FastRestore:Fix windows build 2020-04-28 19:12:27 -07:00
Jingyu Zhou
7d59e53349 Consolidate makePadding() 2020-04-28 15:39:23 -07:00
Meng Xu
f5e8345496 FastRestoreAgent:Use atomicParallelRestore to kick off restore 
Replace the handcrafted version with atomicParallelRestore actor
which is simulation tested
 2020-04-27 22:15:00 -07:00
Meng Xu
d4509090d4 Generate fastrestore_agent symlink in CMake 2020-04-23 07:55:44 -07:00
A.J. Beamon
af4e0088ba
Merge pull request #2896 from tclinken/atomically-update-dependent-knobs 
Atomically update dependent knobs
 2020-04-08 15:00:49 -07:00
tclinken
52860043c9 Merge remote-tracking branch 'origin' into atomically-update-dependent-knobs 2020-04-08 12:26:21 -07:00
Markus Pilman
7ba2cec9ef Merge branch 'master' of github.com:apple/foundationdb into features/no-make 2020-04-08 10:23:46 -07:00
Markus Pilman
d4542dbb5a Delete old build system 2020-04-07 11:03:45 -07:00
Jingyu Zhou
9fb3fb9d82 Add pause/resume for new backups 
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
 2020-04-06 14:29:46 -07:00
Markus Pilman
e4611e8ae4 fix versions.h stupidity 2020-04-06 10:28:55 -07:00
Markus Pilman
8b5780c36c don't include source and binary dir 
This forces users to use include paths from the sources root.

So `#include "Arena.h"` won't work anymore, only
`#include "flow/Arena.h"` will.
 2020-04-06 10:13:49 -07:00
Markus Pilman
bbd2fe62cc Merge branch 'master' of github.com:apple/foundationdb into features/boost70 2020-04-02 09:21:01 -07:00
Meng Xu
4cf796e838
Merge pull request #2891 from jzhou77/backup-cmd 
Add an option for fdbbackup to use new backup system
 2020-04-01 16:12:10 -07:00
tclinken
884e92bb49 Atomically update dependent knobs 2020-04-01 15:18:49 -07:00
Jingyu Zhou
906174e3e8 Add an option for fdbbackup to use new backup system 
I.e., "-p", or "--partitioned_log" to enable it. By default, old backup system
is used.
 2020-03-31 14:54:32 -07:00
Jingyu Zhou
3c76722504 Fix valgrind error of invalid memory access 2020-03-30 16:38:22 -07:00
Jingyu Zhou
3c32835cce Fix decoder for unfinished version batch in a log 
A mutation log's version batch data can be split into multiple blocks, and some
of the blocks can be spread across two mutation logs. Thus, the decoder needs
copy unfinished version batch data from previous file progress to the next one.
 2020-03-30 11:34:51 -07:00
Markus Pilman
28cd38913d Merge branch 'master' of github.com:apple/foundationdb into features/boost70 2020-03-27 13:44:10 -07:00
Jingyu Zhou
feedab02a0
Merge pull request #2855 from xumengpanda/mengxu/fr-api-atomicrestore-PR 
Add ApiCorrectnessAtomicRestore workload for the new performant restore
 2020-03-25 18:05:26 -07:00
Meng Xu
a93f13cfd7 Remove redundant restoreRequestDone break in backup.actor 2020-03-25 15:19:46 -07:00
Meng Xu
495afe2e0b Improve how to wati for restore to finish 
Remove default parameter for atomicRestore as suggested in review.
 2020-03-25 13:54:21 -07:00
Jingyu Zhou
7831bec2b0 Address review comments on trace events 2020-03-24 10:54:12 -07:00
Jingyu Zhou
b18f192831 Fix decode bug of missing mutations 
After reading a new block, all mutations are sorted by version again, which
can invalidate previously tuple. As a result, the decoded file will miss some
of the mutations.
 2020-03-20 20:15:09 -07:00
Jingyu Zhou
ca1a4ef9fd Ignore mutation logs of size 0 in converter 2020-03-20 20:15:08 -07:00
Jingyu Zhou
d0a24dd20d Decode out of order mutations in old mutation logs 
In the old mutation logs, a version's mutations are serialized as a buffer.
Then the buffer is split into smaller chunks, e.g., 10000 bytes each. When
writting chunks to the final mutation log file, these chunks can be flushed
out of order. For instance, the (version, chunck_part) can be in the order of
(3, 0), (4, 0), (3, 1). As a result, the decoder must read forward to find all
chunks of data for a version.

Another complication is that the files are organized into blocks, where (3, 1)
can be in a subsequent block. This change checks the value size for each
version, if the size is smaller than the right size, the decoder will look
for the missing chucks in the next block.
 2020-03-20 20:15:08 -07:00
Jingyu Zhou
20df67ee6a Filter partitioned logs with subset relationship 
If a log file's progress is not saved, a new log file will be generated
with the same begin version. Then we can have a file that contains a subset
of contents in another log file. During restore, we should filter out files
that their contents are subset of other files.
 2020-03-20 20:15:08 -07:00
Jingyu Zhou
e15015ee6c Add mutation log version names 
I.e., BACKUP_AGENT_MLOG_VERSION for 2001 and PARTITIONED_MLOG_VERSION for 4110.
 2020-03-20 20:13:38 -07:00
Evan Tschannen
303df197cf Merge branch 'release-6.2' 
# Conflicts:
#	CMakeLists.txt
#	bindings/c/test/mako/mako.c
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/NativeAPI.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/Knobs.cpp
#	fdbserver/Knobs.h
#	fdbserver/LogRouter.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/fdbserver.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/flow.vcxproj
#	flow/flow.vcxproj.filters
#	versions.target
 2020-03-06 18:22:46 -08:00
Alex Miller
595dd77ed1 Merge remote-tracking branch 'upstream/release-6.2' into certificate-refresh 2020-03-04 20:25:42 -08:00
Alex Miller
9b5ef3416e Refactor TLSParams into TLSConfig + LoadedTLSConfig 
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.

initTLS now just takes the in-memory bytes and applies them to the ssl
context.

This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
 2020-03-04 20:14:47 -08:00
Evan Tschannen
cdcb816866
Update fdbbackup/backup.actor.cpp 2020-03-04 16:08:45 -08:00
A.J. Beamon
58e621eca1 Invalid knobs or knob values are treated as warnings rather than errors. Apply this change to backup as well. 2020-03-04 15:50:04 -08:00
Evan Tschannen
c11c24b79d removed the fdbrpc version of platform.h 2020-02-28 14:56:10 -08:00
Evan Tschannen
96258b9809 Merge branch 'release-6.2' 
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/FlowTransport.actor.cpp
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistribution.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/StorageMetrics.actor.h
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/KVStoreTest.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/genericactors.actor.cpp
#	flow/serialize.h
 2020-02-21 19:09:16 -08:00
Alex Miller
927cff3317 Report errors on TLS misconfigurations ... or at least try to. 2020-02-20 16:57:29 -08:00
Evan Tschannen
761da5a059 code cleanup 2020-02-19 17:59:45 -08:00
Meng Xu
132f5aa9ba FastRestore:Improve trace name and cosmetic change 2020-02-18 16:41:19 -08:00
Meng Xu
31a6ec34b7 Merge branch 'master' into mengxu/fast-restore-agent-PR 2020-02-18 16:17:59 -08:00
mpilman
3a1e878a9b Upgrade to boost 1.72 2020-02-14 18:10:13 -08:00
Jingyu Zhou
e3061ab713 Use ArenaReader to deserialize MutationRef 2020-02-13 10:28:08 -08:00