apple/foundationdb
1
0
Fork 0
mirror of https://github.com/apple/foundationdb.git synced 2025-06-01 02:37:02 +08:00
Code Issues Packages Projects Releases Wiki Activity
14,449 Commits 35 Branches 318 Tags
Commit Graph

80 Commits

Author SHA1 Message Date
FDB Formatster
df90cc89de apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-10 10:18:07 -08:00
A.J. Beamon
aaf0a9aa7b Merge branch 'release-6.3' into merge-release-6.3-into-master 
# Conflicts:
#	build/docker-compose.yaml
#	cmake/ConfigureCompiler.cmake
#	fdbclient/FileBackupAgent.actor.cpp
#	fdbrpc/AsyncFileCached.actor.h
#	fdbrpc/IAsyncFile.h
#	fdbrpc/IRateControl.h
#	fdbrpc/simulator.h
#	fdbserver/KeyValueStoreSQLite.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbservice/ServiceBase.cpp
 2021-02-08 12:58:34 -08:00
Christophe Chevalier
3fc8870ce3 winbuild: fix duplicate definition of Codec<ERestoreState>::pack and unpack in Debug build 
- defined twice in FileBackupAgent.actor.cpp and BackupAgent.actor.h
- only fails when building in Debug ?
 2021-01-30 19:03:25 +01:00
Jon Fu
51db9a7e0a add static method to access backup pause key instead of constructing it manually 2020-11-06 14:03:29 -05:00
sfc-gh-tclinkenbeard
cf4c8e375f Merge remote-tracking branch 'origin/release-6.3' into merge 2020-10-29 22:15:41 -07:00
Jingyu Zhou
e183a10c8c Add "waitForDestUID" flag for abortBackup 2020-10-20 22:55:31 -07:00
sfc-gh-tclinkenbeard
3a03c9bd74 Remove deprecated enumActionResult 2020-09-23 15:14:09 -07:00
sfc-gh-tclinkenbeard
d13a9d3c91 Make EBackupState an enum class 2020-09-23 15:14:09 -07:00
Meng Xu
a3c861b59d
Merge pull request #3777 from apple/release-6.3 
Merge Release 6.3 into master
 2020-09-14 10:32:50 -07:00
Meng Xu
e69ee87f33 Refactor blob failure to simulateBlobFailure function 2020-09-09 10:54:21 -07:00
Jon Fu
00c77ba2b4 Added beginVersion cmd line option and addressed code review comments 2020-08-28 14:29:22 -04:00
Jon Fu
ae999aa118 Merge branch 'master' of https://github.com/apple/foundationdb into jfu-incremental-backup-only 2020-08-19 16:36:47 -04:00
Jon Fu
21635f8a28 update backup restore for local testing 2020-08-04 15:48:43 -04:00
Evan Tschannen
a49cb41de7 Merge branch 'release-6.3' 
# Conflicts:
#	CMakeLists.txt
#	cmake/ConfigureCompiler.cmake
#	fdbserver/Knobs.cpp
#	fdbserver/StorageCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	flow/ThreadHelper.actor.h
#	flow/serialize.h
#	tests/CMakeLists.txt
 2020-07-29 00:31:55 -07:00
Jon Fu
2d09d2f68e re-ordered parameters to fit existing test workloads 2020-07-28 12:39:35 -04:00
Jon Fu
71a1b2cb0d first attempt at making restore pipeline incremental only 2020-07-28 12:39:35 -04:00
Jon Fu
3c62651ad6 initial commit to introduce incremental backup only (ignore base snapshot) 2020-07-28 12:39:35 -04:00
Evan Tschannen
d3093f50bb attempt to get rid of compiler warnings 2020-07-24 12:24:42 -07:00
Jingyu Zhou
90654628fb
Merge pull request #3490 from apple/release-6.3 
Merge Release 6.3
 2020-07-10 12:54:37 -07:00
Oleg Samarin
4c9df78076 dstonly 2020-07-10 15:13:42 +03:00
Jingyu Zhou
3ef341b157
Merge pull request #3436 from oleg68/osamarin5 
Speed up sync between Primary and DR
 2020-07-04 20:21:45 -07:00
Meng Xu
22f7f804b8 Merge branch 'release-6.3' into mengxu/merge-6.3-PR 2020-06-28 11:19:39 -07:00
Oleg Samarin
d887d5dd70 Increased CopyLogRange performance 2020-06-26 13:34:35 +03:00
Meng Xu
198696bc1e Move transformRestoredDatabase from server to client 
AtomicRestore workload turns out to rely on the FileBackupAgent
client. Keeping transformRestoredDatabase in server makes linking harder.
 2020-06-23 15:48:43 -07:00
Meng Xu
2fcef90cbc FastRestore:Add addPrefix and removePrefix feature 2020-06-18 11:58:42 -07:00
Meng Xu
d7a8e554ed FastRestore:Test for addPrefix and removePrefix 
Add the skeleton code to test;
addPrefix and removePrefix are all empty for now.
 2020-06-17 20:36:53 -07:00
sfc-gh-tclinkenbeard
99bf993815 Replace BOOST_NOEXCEPT with noexcept 2020-06-09 22:39:19 -07:00
Meng Xu
e7fdd614fe Remove assert on error_code_key_not_found in previous commit 
Correctnss shows the assert does not hold in all scenarios
 2020-06-08 19:02:56 -07:00
Meng Xu
5022566b35 Validate if key_not_found error ever happens 2020-06-08 16:59:00 -07:00
tclinken
aca995d2c5 Removed dead backup agent code 2020-05-01 14:46:59 -07:00
Meng Xu
3f510d0653
Merge pull request #3036 from jzhou77/backup-cmd 
Several bug fixes for new backups
 2020-04-29 16:34:59 -07:00
Jingyu Zhou
9da76c35ea Fix a memory corruption error 
The backup container URL should be Key instead of KeyRef.
 2020-04-29 15:55:34 -07:00
Jingyu Zhou
7d59e53349 Consolidate makePadding() 2020-04-28 15:39:23 -07:00
Meng Xu
f5e8345496 FastRestoreAgent:Use atomicParallelRestore to kick off restore 
Replace the handcrafted version with atomicParallelRestore actor
which is simulation tested
 2020-04-27 22:15:00 -07:00
Jingyu Zhou
9bfc5bbea8 Check RestorableFileSet's key ranges in simulation 
Ranges written in the manifest file should match with actual file content.
 2020-04-21 12:55:40 -07:00
Jingyu Zhou
9fb3fb9d82 Add pause/resume for new backups 
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
 2020-04-06 14:29:46 -07:00
Jingyu Zhou
feedab02a0
Merge pull request #2855 from xumengpanda/mengxu/fr-api-atomicrestore-PR 
Add ApiCorrectnessAtomicRestore workload for the new performant restore
 2020-03-25 18:05:26 -07:00
Meng Xu
1ba11dc74b Apply clang format 2020-03-25 11:20:17 -07:00
Meng Xu
120272f025 Change unlockDB from RestoreMaster to Agent 2020-03-25 11:04:49 -07:00
Meng Xu
ca8966a28b Move lockDB into submitRestore request from restore worker 
AtomicRestore needs to lock DB before we start the restore worker.
So we cannot lock DB in restore worker with a different randomUID.
 2020-03-24 23:39:35 -07:00
Meng Xu
241c2703c8 Fix atomicParallelRestore interface 2020-03-24 17:00:55 -07:00
Meng Xu
80d62f3cb8 Fix:Add atomicParallelRestore to header 2020-03-24 16:28:08 -07:00
Meng Xu
81f7181c9e Refactor submitParallelRestore function into FileBackupAgent 2020-03-24 14:44:55 -07:00
Meng Xu
5584884c12 Refactor parallelRestoreFinish function into FileBackupAgent 2020-03-24 14:15:15 -07:00
Jingyu Zhou
44c1996950 Change all worker started to be set after all workers updated a key 
Previously, all worker started is set to be when saved log versions are higher.
However, saving the versions can be wrong, as the worker is not guaranteed to
write to the right container. For instance, if the watch is triggered later,
then mutation logs are written to previous containers. So we need to ensure the
right container is ready -- all workers have acknowledged seeing the container.
 2020-03-22 16:40:12 -07:00
Jingyu Zhou
38def426f4 Add a flag to submitBackup for partitioned log 
This is to distinguish with old workloads so that they can work in simulation.
 2020-03-20 20:15:08 -07:00
Jingyu Zhou
e9287407d6 Backup worker updates latest log versions in BackupConfig 
If backup worker is enabled, the current epoch's worker of tag (-2,0) will be
responsible for monitoring the backup progress of all workers and update the
BackupConfig with the latest saved log version, which is the minimum version
of all tags.

This change has been incorporated in the getLatestRestorableVersion() so that
it is transparent to clients.
 2020-03-20 20:15:08 -07:00
Jingyu Zhou
35aafefb89 Consolidate StringRefReader classes 
Fix a compiler error of unused variable too.
 2020-03-20 20:13:38 -07:00
Jingyu Zhou
5a602f58e8 Start backup with a wait on all backup workers running 
This wait is to make sure that backup workers are already saving mutations so
that no mutations are missed. The idea is that the CLI sets a "backupStartedKey"
in the database and waits for allWorkerStarted() key of the backup to be set.

Backup workers monitor the changes to the "backupStartedKey" and start logging
mutations. Additionally, backup worker for Tag(-2,0) monitors all other workers
have started (checking their saved progress version is larger than the backup's
start version), and then sets the allWorkerStarted() key for the backup.
 2020-01-31 19:29:09 -08:00
Jingyu Zhou
7f7ec99170 Serialize and deserialize new backup files 
The BackupWorker writes files that can be read by FileConverter. Move
StringRefReader to the header file for reuse in FileConverter.
 2020-01-22 19:38:46 -08:00