apple/foundationdb
1
0
Fork 0
mirror of https://github.com/apple/foundationdb.git synced 2025-05-15 02:18:39 +08:00
Code Issues Packages Projects Releases Wiki Activity
9,656 Commits 35 Branches 318 Tags
Commit Graph

134 Commits

Author SHA1 Message Date
A.J. Beamon
35c18ac60a Improvements to expiration. Encode throttles with auto/manual and priority in the key to support throttling the same tag with different values in these parameters. 2020-04-23 20:50:40 -07:00
A.J. Beamon
dfec896438 Enforce a throttle limit. Don't count transaction tags on RK if the proxy has updated us in a while. 2020-04-17 11:48:02 -07:00
A.J. Beamon
78d48a0dad Merge branch 'master' into transaction-tagging 
# Conflicts:
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/Knobs.h
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/fdbclient.vcxproj
#	fdbserver/MasterProxyServer.actor.cpp
 2020-04-17 09:23:18 -07:00
Alex Miller
c8ab69cce6 Fix incorrect if/ASSERT logic from last minute cleanups. 
My test code serialized UIDs and Tags, so that I could compare and
verify I got the same thing, and I undid this right before committing,
but forgot to also change the `if` to not rely on UIDs being first also.
 2020-04-12 19:38:51 -07:00
Alex Miller
db071256db Merge remote-tracking branch 'upstream/master' into reduce-txn-state-store-size 2020-04-12 18:06:10 -07:00
A.J. Beamon
ebeca10bce Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients. 2020-04-09 16:55:56 -07:00
A.J. Beamon
36da61dd9c Merge branch 'master' into transaction-tagging 
# Conflicts:
#	fdbclient/NativeAPI.actor.cpp
#	fdbclient/vexillographer/fdb.options
 2020-04-07 21:12:14 -07:00
Jingyu Zhou
60407bdee3 Use LiteralStringRef for backup paused key 2020-04-07 16:02:25 -07:00
Jingyu Zhou
9fb3fb9d82 Add pause/resume for new backups 
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
 2020-04-06 14:29:46 -07:00
Alex Miller
6078fd1b18 Convert UID to Tag in keyServers to reduce txnStateStore size 2020-04-05 14:30:09 -07:00
A.J. Beamon
2336f073ad Checkpointing a bunch of work on throttles. Rudimentary implementation of auto-throttling. Support for manual throttling via fdbcli. Throttles are stored in the system keyspace. 2020-04-03 15:24:14 -07:00
Meng Xu
60f6edc3b5
Merge pull request #2860 from zjuLcg/report-conflicting-key-roll-forward 
Report conflicting key roll forward
 2020-03-30 17:33:56 -07:00
Meng Xu
113d0fb48b Remove incorrect assertion 2020-03-27 12:13:30 -07:00
Andrew Noyes
289487559d Revert "Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key"" 
This reverts commit 804fe1b22e0a5b7f0d8fe87fe86881bfe0928546.
 2020-03-24 18:11:15 -07:00
Balachandar Namasivayam
804fe1b22e Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key" 
This reverts commit 648dc4a933e0f606de07e2b855c62a9ed828cd3a, reversing
changes made to 487d131b38d12f8e766985c77b49e60234acac9e.
 2020-03-19 21:34:28 -07:00
chaoguang
0094293d50 add const vars 2020-03-11 23:11:49 -07:00
chaoguang
d1c56d3b57 add constant KeyRefs in SystemData 2020-03-11 12:25:50 -07:00
chaoguang
7a76e9556d Merge remote-tracking branch 'upstream/master' into report-conflicting-key 2020-03-04 11:24:39 -08:00
Meng Xu
a12a161fb3 Merge branch 'master' into mengxu/fast-restore-pipeline-PR 2020-02-18 14:49:52 -08:00
Meng Xu
c603b20e7e FastRestore:Resolve review comments 2020-02-18 14:08:27 -08:00
Jingyu Zhou
5a602f58e8 Start backup with a wait on all backup workers running 
This wait is to make sure that backup workers are already saving mutations so
that no mutations are missed. The idea is that the CLI sets a "backupStartedKey"
in the database and waits for allWorkerStarted() key of the backup to be set.

Backup workers monitor the changes to the "backupStartedKey" and start logging
mutations. Additionally, backup worker for Tag(-2,0) monitors all other workers
have started (checking their saved progress version is larger than the backup's
start version), and then sets the allWorkerStarted() key for the backup.
 2020-01-31 19:29:09 -08:00
Meng Xu
141609e80a FastRestore:Improve code style and fix typos 2020-01-27 18:13:14 -08:00
Meng Xu
16f9ec45bd Merge branch 'master' into mengxu/fast-restore-pipeline-PR 2020-01-23 20:15:21 -08:00
Jingyu Zhou
1eaea91cb3 Address review comments 2020-01-22 19:42:13 -08:00
Jingyu Zhou
c08a192c75 Add a backup start key 
If the backup key is not set, do not recruit backup workers for old epoches.
 2020-01-22 19:42:13 -08:00
Jingyu Zhou
19d6a889ff Recruit backup workers for old epochs 
If there are unfinished ranges in the old epochs, the new master will recruit
backup workers responsible for finishing these ranges. These workers remains in
the cluster until the next epoch, when it will remove itself.
 2020-01-22 19:38:45 -08:00
Jingyu Zhou
41f0cf2bb5 Add decode function for backup progress 2020-01-22 19:38:45 -08:00
Jingyu Zhou
7da9f47f26 Enable pop from backup workers 
This is still WIP as some edge cases can trigger test failure, most likely due
to not popping mutations by backup workers when epoch ends.
 2020-01-22 19:38:45 -08:00
Meng Xu
bfbf2164c4 FastRestore:Applier buffer data for multiple batches 2020-01-17 17:01:01 -08:00
chaoguang
10719200c3 A hack way to call API through getRange("\xff\xff/conflicting_keys\<start_key>", "\xff\xff/conflicting_keys\<end_key>"). 2020-01-06 11:22:11 -08:00
negoyal
d46c7ded59 Merge remote-tracking branch 'origin/master' into storage-cache-subfeature1 2019-11-14 17:52:22 -08:00
negoyal
a4a0bf18f9 Merging with Master. 2019-11-12 13:01:29 -08:00
Meng Xu
630c29d160 FastRestore:resolve review comments 
1) wait on whenAtLeast;
2) Put BigEndian64 into the function call and the decoder to prevent
future people from making the same mistake.
 2019-11-11 17:00:16 -08:00
Meng Xu
eb67886b75 FastRestore:Move comment to func definition 
Resolve review comments.
 2019-11-11 15:10:27 -08:00
Meng Xu
58aa6711e4 FastRestore:ApplyToDB:BugFix:Serialize integer as bigEndian to ensure lexico order 2019-11-03 17:26:07 -08:00
Andrew Noyes
b7b5d2ead3 Remove several nonsensical const uses 
These seem to be all the ones that clang's -Wignored-qualifiers
complains about
 2019-10-26 14:30:34 -07:00
Jon Fu
f4237ebfff Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed 2019-10-16 11:32:16 -07:00
Meng Xu
71509a5157 FastRestore:Applier:applyToDB:Clang format 2019-10-10 17:36:38 -07:00
Meng Xu
84b5a5525f FastRestore:Add restoreApplierKeys 2019-10-10 17:18:34 -07:00
Jon Fu
471e283128 Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed 2019-09-18 11:49:07 -07:00
Evan Tschannen
8fbd90e2f6
Merge pull request #1985 from xumengpanda/mengxu/storage-engine-switch-PR-v2 
Graceful storage engine migration
 2019-09-09 13:51:53 -07:00
Meng Xu
c2355f721e Merge branch 'master' into mengxu/performant-restore-PR 2019-09-04 17:11:42 -07:00
Meng Xu
d160810662 FastRestore:Resolve review comments 2019-09-04 16:48:43 -07:00
Jon Fu
c908c6c1db added command to fdbcli and changes to SystemData and ManagementAPI 2019-08-27 14:39:43 -07:00
Meng Xu
e6284684f0 StorageEngineSwitch:Always remove wrong storeType SS 
In the old logic of switching storage engines, it marks a storage server
with wrong store type as undesired even though this can lead to no healthy team.

In the first version of the new storage engine switch, we mimic the same logic
of the old version.
 2019-08-13 14:59:46 -07:00
Meng Xu
a588710376 StorageEngineSwitch:Graceful switch 
When fdbcli change storeType for storage engines,
we switch the store type of storage servers one by one gracefully.
This avoids recruiting multiple storage servers on the same process,
which can cause OOM error.
 2019-08-12 17:37:52 -07:00
Jingyu Zhou
4a63de16e9
Merge pull request #1945 from xumengpanda/mengxu/tLog-code-read-v2 
Add comments to DiskQueue and tLog
 2019-08-08 13:24:32 -07:00
Meng Xu
7ff46e6772 Merge branch 'master' into mengxu/performant-restore-PR 2019-08-07 20:31:56 -07:00
Evan Tschannen
ba54508c47 code cleanup 2019-08-06 16:30:30 -07:00
Meng Xu
3b54363780 FastRestore:Apply Clang-format 2019-08-01 18:09:12 -07:00