apple/foundationdb
1
0
Fork 0
mirror of https://github.com/apple/foundationdb.git synced 2025-06-02 19:25:52 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,000 Commits 35 Branches 318 Tags
Commit Graph

181 Commits

Author SHA1 Message Date
Meng Xu
6bce67ca75 FastRestore:Apply clang-format 2020-04-01 21:27:54 -07:00
Meng Xu
c69c959428 FastRestore:Fix:It is legal for a backup key not exist in DB 2020-03-31 22:02:17 -07:00
Meng Xu
25e96a13d3 FastRestore:Fix clearrange on a key mistakenly clear other keys 2020-03-31 17:45:19 -07:00
Meng Xu
212dadc2a1 Fix bug in add mutation on applier 
For clear range mutation, we may clear the right boundary key which should not be cleared.
 2020-03-31 16:51:08 -07:00
Meng Xu
33c4be9c42 Improve debug message for debug mutations 2020-03-31 16:00:51 -07:00
Jingyu Zhou
65e3b9192e Add an assert for probably dead code 2020-03-28 21:19:47 -07:00
Meng Xu
13f343ec96 Resolve minor review comment 2020-03-28 16:03:01 -07:00
Jingyu Zhou
196127fb92 Address review comments 2020-03-23 14:15:36 -07:00
Jingyu Zhou
fea6155714 StagingKey uses mutation instead of a vector of mutations for each log version 
Because each log version contains commit version and subsequence number, each
key can only have one mutation for its log version. This simplifies
StagingKey::add() a lot.
 2020-03-20 20:15:09 -07:00
Jingyu Zhou
799f0b4b0e Small code refactor 2020-03-20 20:15:09 -07:00
Jingyu Zhou
ab0b59b0c3 Add subsequence number to restore loader & applier 
The subsequence number is needed so that mutations of the same commit version
number, but from different partitioned logs can be correctly reassembled in
order.

For old backup files, the sub number is always 0. For partitioned mutation
logs, the actual sub number is used. For range files, the sub number is always
0.
 2020-03-20 20:13:38 -07:00
Andrew Noyes
c3b67c0c63 Fix OPEN_FOR_IDE build 2020-03-03 11:32:43 -08:00
Meng Xu
e6457ba0d5 FastRestore:Correct type for imcompleteStagingKeys 2020-03-02 11:33:07 -08:00
Meng Xu
2520e8d44c FastRestore:Use more concise code as suggested in review 2020-03-01 22:32:36 -08:00
Meng Xu
01c1a15caf FastRestore:Applier:Limit fetch keys number in a txn in getAndComputeStagingKeys 2020-02-28 16:53:36 -08:00
Meng Xu
fe8b8bbbff FastRestore:Change vb state to class from enum 2020-02-27 20:15:25 -08:00
Meng Xu
d77177367c FastRestore:Track each ongoing version batch progress state for applier and loader roles 2020-02-27 19:47:22 -08:00
Meng Xu
fbb6e8f39d FastRestore:Create low memory situation in simulation on purpose 2020-02-26 14:54:38 -08:00
Meng Xu
06495b90ae FastRestore:Loader:Use isSchedulable to guard OOM 
And trigger delayed actors that are blocked on memory to recheck memory.
 2020-02-26 14:35:05 -08:00
Meng Xu
a354f6ffa2 FastRestore:Applier:Use isSchedulable to guard OOM 2020-02-26 14:12:56 -08:00
Meng Xu
ca726fc68e FastRestore:Introduce OOM protection 
An actor is schedulable to run if the current worker has enough resourc, i.e.,
the worker's memory usage is below the threshold;
Exception: If the actor is working on the current version batch, we have to schedule
the actor to run to avoid dead-lock.
Future: When we release the actors that are blocked by memory usage, we should release them
in increasing order of their version batch.
 2020-02-26 14:09:18 -08:00
Meng Xu
fbf5020af9 FastRestore:Applier:Add fetchKeys counter 2020-02-26 11:37:40 -08:00
Meng Xu
6bd4703a9f FastRestore:Resolve review comments 2020-02-20 14:27:34 -08:00
Meng Xu
d5d26f589f FastRestore:Cosmetic change to improve code readability 2020-02-19 15:43:51 -08:00
Meng Xu
7897b1658f FastRestore:Applier:Rename new apply actor names 2020-02-19 15:29:32 -08:00
Meng Xu
e4258d73f5 FastRestore:Applier:Remove applying actors that do not have good perf 2020-02-19 15:27:59 -08:00
Meng Xu
fe75a4cafb FastRestore:Apply clang-format 2020-02-19 15:22:52 -08:00
Meng Xu
03f699f2f9 Merge branch 'master' into mengxu/fast-restore-applier-multi-applying-PR 2020-02-19 15:22:33 -08:00
Meng Xu
94d799552e FastRestore:Apply clang-format against master 2020-02-18 16:41:59 -08:00
Meng Xu
132f5aa9ba FastRestore:Improve trace name and cosmetic change 2020-02-18 16:41:19 -08:00
Meng Xu
31a6ec34b7 Merge branch 'master' into mengxu/fast-restore-agent-PR 2020-02-18 16:17:59 -08:00
Meng Xu
c603b20e7e FastRestore:Resolve review comments 2020-02-18 14:08:27 -08:00
Meng Xu
c34a69df32 FastRestore:Applier:Remove unused func 2020-02-13 23:06:56 -08:00
Meng Xu
3e2c19630a FastRestore:Applier:atomicOp can work on an empty key 2020-02-13 23:05:54 -08:00
Meng Xu
b57583a504 FastRestore:Applier:Handle multiple gets in parallel 2020-02-13 23:05:31 -08:00
Meng Xu
53f427c319 FastRestore:Applier:fix getAndComputeStagingKeys 2020-02-13 22:11:30 -08:00
Meng Xu
0d668ea0c3 FastRestore:Applier:Add more trace for perf tracking 2020-02-13 15:50:10 -08:00
Meng Xu
0b27786811 FastRestore:Applier:Minor change for clang-format 2020-02-13 13:17:32 -08:00
Meng Xu
b5e60585aa FastRestore:Applier:Fix precompute mutation result 2020-02-13 12:57:47 -08:00
Meng Xu
b1b44d4477 FastRestore:Applier:Handle CompareAndClear atomicOp 2020-02-13 11:11:29 -08:00
Meng Xu
58dad5373b FastRestore:Applier:Handle CompareAndClear atomicOp 2020-02-13 11:06:20 -08:00
Meng Xu
d3c01763d9 FastRestore:Applier:Handle version stamped key values 2020-02-13 10:48:36 -08:00
Meng Xu
b008df97eb FastRestore:Applier:Multiple set-clear mutations at same version 2020-02-13 10:13:46 -08:00
Meng Xu
238b2cb8e4 FastRestore:Applier:Fix various bugs 
1. segmentation error
2. there exist mutations that is not set or clear or atomicOp, precompute result should ignore them.
 2020-02-13 10:00:23 -08:00
Meng Xu
acf34319c1 FastRestore:Applier:Precompute mutations and apply in parallel 
Precompute mutations received by an applier;
Only apply the final result to the destination DB;
Execute multiple txns in parallel to apply final results to the destination DB.
 2020-02-12 22:47:48 -08:00
Meng Xu
2bc82ffd70 FastRestore:Applier:Store received mutation by key 2020-02-12 14:12:38 -08:00
Meng Xu
c0f75d77b1 FastRestore:Applier:Intro StagingKey struct 2020-02-12 13:57:18 -08:00
Meng Xu
3e6bbe9e5b FastRestore:Applier:Use real size for atomic op 2020-02-11 15:51:32 -08:00
Meng Xu
cda8fc189e FastRestore:AtomicOp:Intro weighted size for atomicOp 
atomicOp has an amplified performance overhead to the cluster,
for example, an ADD operation can be small, but SS has to load
the value to do the operation and the value can be large.
 2020-02-11 12:48:05 -08:00
Meng Xu
e76b6d824a FastRestore:Assign priority to actors to prioritize vb work 
When we pipeline multiple version batches, we should prevent a later
version batch from blocking the earlier version batch by consuming
CPU resources.

To achive the above, we should assign higher priority to actors
in later phases in a version batch.

Because restore master will not invoke an actor at a later phase unless
the actors at the earlier phases have been finished. This priority assignment
will not cause dead lock.
 2020-02-10 20:29:23 -08:00