1
0
mirror of https://github.com/apple/foundationdb.git synced 2025-06-01 02:37:02 +08:00

73 Commits

Author SHA1 Message Date
Suraj Gupta
3fe7a9f553 More fixes 2021-12-10 16:43:58 -06:00
Evan Tschannen
951bc4acd7 fix: do not call better master exists until the long lived stateless processes have settled into their desired locations 2021-12-06 13:12:27 -08:00
Evan Tschannen
243927c964 added a knob 2021-12-03 10:31:51 -08:00
Steve Atherton
bed25f9571 Delay prioritized eviction of updated pages until after commit completes. 2021-11-28 21:03:44 -08:00
Steve Atherton
508429f30d
Redwood chunked file growth and low priority IO starvation prevention ()
* Redwood files now growth in large page chunks controlled by a knob to reduce truncate() calls for expansion.   PriorityMultiLock has limit on consecutive same-priority lock release.  Increased Redwood max priority level to 3 for more separation at higher BTree levels.

* Simulation fix, don't mark certain IO timeout errors as injected unless the simulated process has been set to have an unreliable disk.

* Pager writes now truncate gradually upward, one chunk at a time, in response to writes, which wait on only the necessary truncate operations.   Increased buggified chunk size because truncate can be very slow in simulation.

* In simulation, ioTimeoutError() and ioDegradedOrTimeoutError() will wait until at least the target timeout interval past the point when simulation is sped up.

* PriorityMultiLock::toString() prints more info and is now public.

* Added queued time to PriorityMultiLock.

* Bug fix to handle when speedUpSimulation changes later than the configured time.

* Refactored mutation application in leaf nodes to do fewer comparisons and do in place value updates if the new value is the same size as the old value.

* Renamed updatingInPlace to updatingDeltaTree for clarity.  Inlined switchToLinearMerge() since it is only used in one place.

* Updated extendToCover to be more clear by passing in the old extension future as a parameter.  Fixed initialization warning.
2021-11-12 13:47:07 -08:00
Daniel Smith
66520eb1c1 Utilize read types to do selective throttling 2021-11-10 11:51:04 -05:00
Tao Lin
fdb3b72e35 Introduce GetRangeAndFlatMap to push computations down to FDB
Re-introduce 
2021-11-09 13:52:28 -08:00
Tao Lin
586cc3b102
Revert "Introduce GetRangeAndFlatMap to push computations down to FDB" 2021-11-04 08:46:56 -07:00
Tao Lin
0853661d13 Introduce getRangeAndHop to push computations down to FDB 2021-11-03 13:21:16 -07:00
Xiaoxi Wang
1a2a838df3 add knob 2021-10-27 09:08:37 -07:00
Evan Tschannen
2208b04174
Merge pull request from sfc-gh-etschannen/blob_full_clean
Blob Granules V0
2021-10-26 09:57:35 -07:00
Lukas Joswiak
c96f560cbe Verify rollback of a single version in simulation, other small fixes 2021-10-25 12:03:22 -07:00
Josh Slocum
0ff8ddc2b6 Merge branch 'master' into blob_full_clean 2021-10-25 13:38:48 -05:00
Steve Atherton
d153519188
Merge pull request from sfc-gh-jslocum/ss_ebrake_streaming_fix
Fixes to ss e-brake, tlog streaming, and their interaction
2021-10-22 10:46:17 -07:00
Josh Slocum
773886515e Merge branch 'feature-range-feed' into blob_full_clean 2021-10-22 11:07:51 -05:00
Zhe Wu
0cf829ef91 Reduce restore error message 2021-10-20 14:02:48 -07:00
Josh Slocum
8dd7f8f447 Fixes to ss e-brake, tlog streaming, and their interaction 2021-10-20 10:48:29 -05:00
Josh Slocum
912ef76f1c cleanup before merge 2021-10-18 17:11:14 -05:00
A.J. Beamon
507a09893c
Add ClientCount to ClusterControllerMetrics () 2021-10-17 20:47:11 -07:00
Josh Slocum
5f0ec0612a Merge branch 'feature-range-feed' into blob_full 2021-10-13 15:44:35 -05:00
Zhe Wu
c07a07dbbe Take uptime into account when making failover decision 2021-10-07 11:19:34 -07:00
Zhe Wu
62197faa46 Add more comments to the code 2021-10-07 11:19:34 -07:00
Zhe Wu
c0fbe5471f Implement the core logic of grey failure triggered failover 2021-10-07 11:19:34 -07:00
Suraj Gupta
4d54669ccd Recruit the blob workers via blob manager.
In this PR, the blob manager now recruits blob workers
(via communication with the cluster controller). Blob workers
are onboarded as blob worker processes enter the cluster.
2021-10-04 11:07:08 -04:00
Suraj Gupta
5fa6c687d6 Add blob manager as a singleton. 2021-09-23 10:45:37 -04:00
Suraj Gupta
72edcd8d73 Address PR comments.
Revert knob name change, fix comparison between new and old
recruitments, and get rid of empty `if` block.
2021-09-22 16:56:34 -05:00
Suraj Gupta
0b6fecddbc Refactor logic for recruiting singletons.
This commit refactors the logic for recruiting singletons,
which is done by the ClusterController. This allows for far
easier additions of new singletons in the future, and also
cleans up the code.

Also, the logic for recruiting DD was changed to mirror
the logic for recruiting RK. Although the logic for RK
allows there to be many RKs existing at once, the moveKeysLock
mechanism used by DD still prevents multiple DDs existing at once.
2021-09-22 16:56:18 -05:00
Suraj Gupta
6533678f0d Address PR comments.
Revert knob name change, fix comparison between new and old
recruitments, and get rid of empty `if` block.
2021-09-20 14:26:42 -05:00
Suraj Gupta
fe098b3b11 Refactor logic for recruiting singletons.
This commit refactors the logic for recruiting singletons,
which is done by the ClusterController. This allows for far
easier additions of new singletons in the future, and also
cleans up the code.

Also, the logic for recruiting DD was changed to mirror
the logic for recruiting RK. Although the logic for RK
allows there to be many RKs existing at once, the moveKeysLock
mechanism used by DD still prevents multiple DDs existing at once.
2021-09-20 14:26:42 -05:00
Josh Slocum
c2d1d1704f Merge branch 'feature-range-feed' into blob_full 2021-09-10 11:21:52 -05:00
Josh Slocum
eb76343dfb Added blob granule reassignment and splitting 2021-09-08 14:09:14 -05:00
Steve Atherton
be440ab954
Merge pull request from FuhengZhao/RedwoodHistogram
Redwood local histograms
2021-08-26 12:05:44 -07:00
Josh Slocum
5259af787d Switched blob implementation to use backup container 2021-08-24 13:47:47 -05:00
Fuheng Zhao
b65a66fab7 log redwood histogram seperatly 2021-08-24 09:57:39 -07:00
Neethu Haneesha Bingi
02b3ed3ff1 Adding deadline option to rocksdb calls. 2021-08-19 14:11:28 -07:00
Neethu Haneesha Bingi
fbb393f998 Added readrange timeoout check and rocksdb read deadline option. 2021-08-19 14:11:28 -07:00
Neethu Haneesha Bingi
24ac173c95 Adding ReadRangeAction timeout, returning error and using timer_monotonic changes. 2021-08-19 14:11:28 -07:00
Neethu Haneesha Bingi
01e85610ab Cancelling the timedout reads with rocksdb storage. 2021-08-19 14:11:28 -07:00
Daniel Smith
45b40addb7
Merge pull request from neethuhaneesha/eagerReadsDisable
Disabling option for removing eagerReads for ClearRange mutations.
2021-08-19 12:39:30 -04:00
Neethu Haneesha Bingi
c45daf6f51 Disabling option for removing eagerReads for ClearRange mutations. 2021-08-13 01:26:50 -07:00
Xiaoxi Wang
a97570bd06 solve mis-spelling, trace log and format problems 2021-08-11 18:26:00 -07:00
Josh Slocum
921a2cfca1 Merge branch 'feature-range-feed' into blob_full 2021-08-10 11:25:48 -05:00
Josh Slocum
e83b27387a POC worker that dumps fdb range to an s3 object 2021-08-06 11:01:37 -05:00
Xiaoxi Wang
fd74a16f35 format code 2021-08-02 14:24:20 -07:00
Xiaoxi Wang
2df0474fec merge master 2021-08-02 11:58:35 -07:00
Daniel Smith
a26dbba66f Add RocksDB metrics 2021-07-30 13:27:43 -04:00
Xiaoxi Wang
10c82b422f merge master branch 2021-07-28 14:19:46 -07:00
Dan Lambright
55094bdd00
Merge branch 'master' into expired-forward 2021-07-26 19:58:29 -04:00
Dan Lambright
ae424f1195 rebase 2021-07-26 10:55:22 -04:00
sfc-gh-tclinkenbeard
6836e49073 Throw error when commitProxy gets stuck 2021-07-19 21:00:28 -07:00