foundationdb

mirror of https://github.com/apple/foundationdb.git synced 2025-05-14 09:58:50 +08:00

History

Ata E Husain Bohra 1520390bc5 Refactor: ClusterController driving cluster-recovery state machine

diff-1: Address Jingyu's review comments
 diff-2: Introduce ClusterRecovery actor to seperate out
         cluster recovery code

At present, cluster recovery process consists of following steps:
1. ClusterController clusterWatchDatabase actor recruits
   master/sequencer process.
2. Sequencer process implements the cluster recovery state machine,
   responsible to recruit all other processes as well restore the
   cluster state.

Patch proposes a scheme where the cluster recovery state machine
is implemented and driven by the ClusterController process instead
of the Sequencer process.

Advantages of the scheme could be:
1. Simplified design where ClusterController recruits "sequencer"
   process like other worker processes compared to current scheme
   where "sequencer" process gets special treatment. In newer scheme
   sequencer is responsible for maintaining/providing
   "committed version" (as expected).
2. ClusterController is responsible for worker processes recruitment,
   the sequencer though orchestrating the recovery state machine, it
   need to reachout to the ClusterController for recruiting worker
   processes etc.

NOTE:
Patch has moved the recovery state machine code from
'sequencer' -> 'cluster-controller' process, however, necessary
updates were done for both functionality as well as performance
improvement reasons.

Next Steps:
Cluster recovery documentation will be updated in near future.

2021-12-22 14:06:27 -08:00

sqlite

fix typo

2021-06-28 11:14:55 +08:00

workloads

Add Minicycle Workload

2021-12-06 15:46:40 -08:00

ApplyMetadataMutation.cpp

Add protection against storage and tlog data deletion when joining a new cluster

2021-11-09 12:29:47 -08:00

ApplyMetadataMutation.h

Refactor ApplyMetadataMutation for better readability

2021-08-24 12:58:14 -07:00

art_impl.h

Fix -Wunused-but-set-variable warnings

2021-12-01 18:15:53 -08:00

art.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

ArtMutationBuffer.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

BackupInterface.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

BackupProgress.actor.cpp

Create RangeResult type alias

2021-05-03 13:14:16 -07:00

BackupProgress.actor.h

Enforce std:: specifier rather than using namespace

2021-09-16 19:40:28 -07:00

BackupWorker.actor.cpp

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

BlobManager.actor.cpp

Expand use of fmt to get rid of %ld usage

2021-11-17 17:03:32 -08:00

BlobManagerInterface.h

Add blob manager as a singleton.

2021-09-23 10:45:37 -04:00

BlobWorker.actor.cpp

Merge pull request #5910 from sfc-gh-jslocum/bg_bindings

2021-12-02 11:40:26 -08:00

ClusterController.actor.cpp

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

ClusterController.actor.h

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

ClusterRecovery.actor.cpp

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

ClusterRecovery.actor.h

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

CMakeLists.txt

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

CommitProxyServer.actor.cpp

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

CompactMap.cpp

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

ConfigBroadcaster.actor.cpp

Remove linear time loop

2021-08-23 14:02:41 -07:00

ConfigBroadcaster.h

Add worker kill unit test

2021-08-23 12:45:42 -07:00

ConfigBroadcastInterface.h

Rename config broadcast interface messages

2021-08-10 11:39:29 -07:00

ConfigDatabaseUnitTests.actor.cpp

Merge rollback message into rollforward message

2021-10-25 12:03:22 -07:00

ConfigFollowerInterface.cpp

Merge rollback message into rollforward message

2021-10-25 12:03:22 -07:00

ConfigFollowerInterface.h

Merge rollback message into rollforward message

2021-10-25 12:03:22 -07:00

ConfigNode.actor.cpp

Add assertions to check rollback

2021-10-25 12:03:22 -07:00

ConfigNode.h

Implement IClosable in OnDemandStore

2021-09-13 12:28:17 -07:00

ConflictSet.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

CoordinatedState.actor.cpp

Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem.

2021-10-22 11:05:18 -07:00

CoordinatedState.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

Coordination.actor.cpp

Coordinator should reply clientInfo when it changes.

2021-12-10 16:37:48 -08:00

CoordinationInterface.h

Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem.

2021-10-22 11:05:18 -07:00

CoroFlow.actor.cpp

Merge remote-tracking branch 'origin/master' into fix-clang-warnings

2021-07-28 12:29:27 -07:00

CoroFlow.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

CoroFlowCoro.actor.cpp

Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use.

2021-07-26 19:55:10 -07:00

DataDistribution.actor.cpp

Fix -Wunused-but-set-variable warnings

2021-12-01 18:15:53 -08:00

DataDistribution.actor.h

consider wiggling when waitUntilHealthy

2021-10-27 09:08:37 -07:00

DataDistributionQueue.actor.cpp

fix: self->shardsAffectedByTeamFailure->moveShard must be called without any waits after getting the destination team or we could miss failure notifications for the storage servers in the destination team

2021-11-23 15:26:38 -08:00

DataDistributionTracker.actor.cpp

Merge pull request #6103 from sfc-gh-ajbeamon/fix-dd-merge-too-soon

2021-12-06 15:01:47 -08:00

DataDistributorInterface.h

Adjust distributorSplitRange order.

2021-11-11 20:28:55 -08:00

DBCoreState.h

Enforce std:: specifier rather than using namespace

2021-09-16 19:40:28 -07:00

DeltaTree.h

Redwood inline same-size value updates (#5925 )

2021-11-10 08:22:57 -08:00

DiskQueue.actor.cpp

Make IClosable interface const-correct

2021-10-18 13:40:47 -07:00

FDBExecHelper.actor.cpp

forgot to save before commit

2021-09-25 16:35:54 -06:00

FDBExecHelper.actor.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

fdbserver.actor.cpp

Convert hyphens to underscores for all prefix-based arguments (e.g. --knob-, --locality-)

2021-12-14 12:01:44 -08:00

GrvProxyServer.actor.cpp

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

IConfigConsumer.cpp

Add UseConfigDB class

2021-06-10 20:57:50 -07:00

IConfigConsumer.h

Update fdbserver/IConfigConsumer.h

2021-06-12 08:04:09 -07:00

IDiskQueue.h

Add FDB_ prefix to BOOLEAN_PARAM macros

2021-07-09 05:42:14 -07:00

IKeyValueContainer.h

Move IKeyValueContainer from flow to fdbserver

2020-11-15 12:23:08 -08:00

IKeyValueStore.h

Make IClosable interface const-correct

2021-10-18 13:40:47 -07:00

IPager.h

Remove explicit tryEvict(x) as it is unused and it is functionally replaced by prioritizeEviction(x).

2021-11-27 03:26:34 -08:00

KeyValueStoreCompressTestData.actor.cpp

Make IClosable interface const-correct

2021-10-18 13:40:47 -07:00

KeyValueStoreMemory.actor.cpp

Make IClosable interface const-correct

2021-10-18 13:40:47 -07:00

KeyValueStoreRocksDB.actor.cpp

KeyValueStoreRocksDB histograms to track latencies

2021-12-21 23:09:46 -08:00

KeyValueStoreSQLite.actor.cpp

Make IClosable interface const-correct

2021-10-18 13:40:47 -07:00

Knobs.h

rebase

2021-07-26 10:55:22 -04:00

LatencyBandConfig.cpp

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

LatencyBandConfig.h

Use LF instead of CRLF in LatencyBandConfig

2021-03-11 08:56:55 -08:00

LeaderElection.actor.cpp

Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem.

2021-10-22 11:05:18 -07:00

LeaderElection.h

Minor leader election code improvements.

2021-09-16 15:34:45 -07:00

LocalConfiguration.actor.cpp

Implement IClosable in OnDemandStore

2021-09-13 12:28:17 -07:00

LocalConfiguration.h

Implement IClosable in OnDemandStore

2021-09-13 12:28:17 -07:00

LogProtocolMessage.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

LogRouter.actor.cpp

Extract methods in LogSystem.h to corresponding cpp file

2021-09-12 14:17:19 -07:00

LogSystem.cpp

Extract methods in LogSystem.h to corresponding cpp file

2021-09-12 14:17:19 -07:00

LogSystem.h

Add protection against storage and tlog data deletion when joining a new cluster

2021-11-09 12:29:47 -08:00

LogSystemConfig.cpp

Extract methods in LogSystemConfig.h to corresponding cpp file

2021-09-12 14:17:19 -07:00

LogSystemConfig.h

fixup! Reformat the code per github's requirement

2021-09-12 14:17:19 -07:00

LogSystemDiskQueueAdapter.actor.cpp

Make IClosable interface const-correct

2021-10-18 13:40:47 -07:00

LogSystemDiskQueueAdapter.h

Make IClosable interface const-correct

2021-10-18 13:40:47 -07:00

LogSystemPeekCursor.actor.cpp

Enforce std:: specifier rather than using namespace

2021-09-16 19:40:28 -07:00

MasterInterface.h

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

masterserver.actor.cpp

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

MetricLogger.actor.cpp

Enforce std:: specifier rather than using namespace

2021-09-16 19:40:28 -07:00

MetricLogger.actor.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

MoveKeys.actor.cpp

merge fix

2021-09-22 16:55:49 -07:00

MoveKeys.actor.h

merge from master

2021-09-22 13:07:38 -07:00

MutationTracking.cpp

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-08-27 17:07:47 -07:00

MutationTracking.h

Enforce std:: specifier rather than using namespace

2021-09-16 19:40:28 -07:00

networktest.actor.cpp

Use fmt to address -Wformat warnings

2021-11-17 14:45:48 -08:00

NetworkTest.h

added a sequence number to streaming replies as a safety backstop against out of order delivery

2021-09-20 13:58:05 -07:00

OldTLogServer_4_6.actor.cpp

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

OldTLogServer_6_0.actor.cpp

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

OldTLogServer_6_2.actor.cpp

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

OnDemandStore.actor.cpp

Make IClosable interface const-correct

2021-10-18 13:40:47 -07:00

OnDemandStore.h

Make IClosable interface const-correct

2021-10-18 13:40:47 -07:00

PaxosConfigConsumer.actor.cpp

Add assertions to check rollback

2021-10-25 12:03:22 -07:00

PaxosConfigConsumer.h

Added PImpl class

2021-08-09 10:04:34 -07:00

ProxyCommitData.actor.h

Enforce std:: specifier rather than using namespace

2021-09-16 19:40:28 -07:00

pubsub.actor.cpp

s/subcription/subscription

2021-09-12 17:33:25 -07:00

pubsub.h

s/subcription/subscription

2021-09-12 17:33:25 -07:00

QuietDatabase.actor.cpp

Merge branch 'master' into blob_full_clean

2021-10-25 13:38:48 -05:00

QuietDatabase.h

Merge branch 'feature-range-feed' into blob_full

2021-10-13 15:44:35 -05:00

RadixTree.h

Avoid calling memcpy with null 2nd argument

2021-06-04 09:48:06 -07:00

Ratekeeper.actor.cpp

cleanup before merge

2021-10-18 17:11:14 -05:00

RatekeeperInterface.h

Reader can read blob worker interface and mapping from db, read files, apply mutations, and return result

2021-08-06 11:02:05 -05:00

RecoveryState.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

Resolver.actor.cpp

Fix -Wunused-but-set-variable warnings

2021-12-01 18:15:53 -08:00

ResolverInterface.h

Implement the core logic of gray network triggered recovery in cluster controller

2021-06-30 11:27:05 -07:00

RestoreApplier.actor.cpp

Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use.

2021-07-26 19:55:10 -07:00

RestoreApplier.actor.h

Fix trace event detail name collision in MutationTracking use cases.

2021-08-24 08:04:19 -07:00

RestoreCommon.actor.cpp

Rename ::TRUE and ::FALSE in BooleanParams to ::True and ::False so as to not conflict with the TRUE and FALSE macros provided by the Windows and MacOS SDKs.

2021-07-17 00:11:40 -07:00

RestoreCommon.actor.h

Move RestoreWorkerInterface into fdbserver

2021-05-30 15:02:33 -07:00

RestoreController.actor.cpp

Enable unused-function warning for clang

2021-11-01 14:18:31 -07:00

RestoreController.actor.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

RestoreLoader.actor.cpp

Enforce std:: specifier rather than using namespace

2021-09-16 19:40:28 -07:00

RestoreLoader.actor.h

Fix many -Wreorder-ctor warnings

2021-07-23 17:33:18 -07:00

RestoreRoleCommon.actor.cpp

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

RestoreRoleCommon.actor.h

Move RestoreWorkerInterface into fdbserver

2021-05-30 15:02:33 -07:00

RestoreUtil.actor.cpp

…

RestoreUtil.h

Move RestoreWorkerInterface into fdbserver

2021-05-30 15:02:33 -07:00

RestoreWorker.actor.cpp

Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem.

2021-10-22 11:05:18 -07:00

RestoreWorker.actor.h

Move RestoreWorkerInterface into fdbserver

2021-05-30 15:02:33 -07:00

RestoreWorkerInterface.actor.cpp

Move RestoreWorkerInterface into fdbserver

2021-05-30 15:02:33 -07:00

RestoreWorkerInterface.actor.h

Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem.

2021-10-22 11:05:18 -07:00

RoleLineage.actor.cpp

Revert "Merge pull request #4802 from sfc-gh-ljoswiak/revert/actor-lineage"

2021-06-04 13:31:55 -07:00

RoleLineage.actor.h

Lazy inititialization

2021-06-04 15:01:18 -07:00

ServerDBInfo.actor.h

Sync cluster ID through ServerDBInfo

2021-11-09 12:29:48 -08:00

ServerDBInfo.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-05 16:31:35 -06:00

SigStack.cpp

Add SIGUSR1 and SIGUSR2 macros for Windows

2021-08-18 18:06:36 +02:00

SimpleConfigConsumer.actor.cpp

Add rollback and rollforward logic to ConfigBroadcaster

2021-10-25 12:03:22 -07:00

SimpleConfigConsumer.h

Added PImpl class

2021-08-09 10:04:34 -07:00

SimulatedCluster.actor.cpp

Merge remote-tracking branch 'origin/master' into add-format-warning

2021-11-17 12:14:01 -08:00

SimulatedCluster.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

SkipList.cpp

Enforce std:: specifier rather than using namespace

2021-09-16 19:40:28 -07:00

SpanContextMessage.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

Status.actor.cpp

Remove some usages of PRId64 by using fmt library

2021-11-30 23:35:36 -08:00

Status.h

Merge branch 'master' into bit-flipping-workload

2021-10-11 16:34:57 -07:00

StorageCache.actor.cpp

Enforce std:: specifier rather than using namespace

2021-09-16 19:40:28 -07:00

StorageMetrics.actor.h

Enforce std:: specifier rather than using namespace

2021-09-16 19:40:28 -07:00

StorageMetrics.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

storageserver.actor.cpp

Adding explicit empty version back to change feeds for now

2021-12-10 10:04:05 -06:00

TagPartitionedLogSystem.actor.cpp

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

TagPartitionedLogSystem.actor.h

Add protection against storage and tlog data deletion when joining a new cluster

2021-11-09 12:29:47 -08:00

template_fdb.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

tester.actor.cpp

Merge remote-tracking branch 'origin/master' into add-format-warning

2021-11-17 12:14:01 -08:00

TesterInterface.actor.h

Abstract the cluster file into a cluster connection record that can be backed by something other than the filesystem.

2021-10-22 11:05:18 -07:00

TLogInterface.h

added a sequence number to streaming replies as a safety backstop against out of order delivery

2021-09-20 13:58:05 -07:00

TLogServer.actor.cpp

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

TSSMappingUtil.actor.cpp

Enforce std:: specifier rather than using namespace

2021-09-16 19:40:28 -07:00

TSSMappingUtil.actor.h

Enforce std:: specifier rather than using namespace

2021-09-16 19:40:28 -07:00

VersionedBTree.actor.cpp

Remove unnecessary temporary objects when appending to vector of pairs

2021-12-01 18:15:16 -08:00

VFSAsync.cpp

Fix many -Wreorder-ctor warnings

2021-07-23 17:33:18 -07:00

VFSAsync.h

Enhance trace event.

2021-11-16 17:39:59 -08:00

WaitFailure.actor.cpp

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

WaitFailure.h

apply clang-format to *.c, *.cpp, *.h, *.hpp files

2021-03-10 10:18:07 -08:00

worker.actor.cpp

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00

WorkerInterface.actor.h

Refactor: ClusterController driving cluster-recovery state machine

2021-12-22 14:06:27 -08:00