apple/foundationdb
1
0
Fork 0
mirror of https://github.com/apple/foundationdb.git synced 2025-05-14 18:02:31 +08:00
Code Issues Packages Projects Releases Wiki Activity
18,915 Commits 35 Branches 318 Tags
Commit Graph

119 Commits

Author SHA1 Message Date
Jingyu Zhou
00b57d4cce Merge branch 'main-67eba5ec7' into vv 
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbclient/StorageServerInterface.h
	fdbserver/CommitProxyServer.actor.cpp
	fdbserver/storageserver.actor.cpp
 2022-03-30 20:05:55 -07:00
Jingyu Zhou
e9659b5dd4 Merge branch 'master-PR-6500' into vv 
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbserver/masterserver.actor.cpp
 2022-03-30 14:53:49 -07:00
A.J. Beamon
6f7adb04a5 Send a version along with GetKeyServerLocationsRequests to allow the proxy to reject missing tenants more quickly. Change the commit proxy tenant map to be unversioned. 2022-03-08 13:33:56 -08:00
A.J. Beamon
3ae98189af Server processes send unknown_tenant responses back to clients, which is meant to be retried after updating the tenant cache. Fix bug where key selectors could be truncated after applying the tenant prefix. 2022-03-06 21:54:21 -08:00
A.J. Beamon
72a34945ce Add the ability to disable tenants. Server processes verify the ID of tenants being read or written. 2022-03-06 21:54:21 -08:00
A.J. Beamon
c99c7cf55f Apply tenant prefixes on the storage server. The commit proxy can serve tenant location requests. Fix some bugs. 2022-03-06 21:54:21 -08:00
A.J. Beamon
5fa9d3e1b7 Add a tenant parameter to read and commit requests. Store a map of all tenants on commit proxy and storage servers. Add an option to require tenant mode. 2022-03-06 21:54:21 -08:00
A.J. Beamon
250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Sreenath Bodagala
cddce79f25
Merge pull request #6359 from sbodagala/version-vector-prototype 
Version vector prototype: Merge with the main branch
 2022-02-08 15:56:52 -05:00
Sreenath Bodagala
72e06369a4 Merge remote-tracking branch 'apple-upstream/main' into version-vector-prototype 2022-02-08 17:47:57 +00:00
Jingyu Zhou
54d96633c0 Do not sent metadata mutations to all resolvers 
Let only resolver 0 to process metadata mutations and generate private mutations.
 2022-02-07 11:49:56 -08:00
Trevor Clinkenbeard
e6b02532a6
Merge pull request #6192 from sfc-gh-tclinkenbeard/improve-commitbatcher-performance 
Make `VectorRefPreserializer` move constructor noexcept
 2022-01-31 09:52:51 -08:00
Dan Lambright
9544379cdf rebase 2022-01-20 11:12:33 -05:00
Ata E Husain Bohra
936bf5336a
Revert "Revert "Refactor: ClusterController driving cluster-recovery state machine" (#6191) 
* Revert "Revert "Refactor: ClusterController driving cluster-recovery state machine""

Major changes includes:
1. Re-revert Sequencer refactor commits listed below (in listed order):
1.a. This reverts commit bb17e194d9c9888e203421290959bd7f2c075d7f.
1.b. This reverts commit d174bb2e06bff01157d16c652073536c54d17f7f.
1.c. This reverts commit 30b05b469c87d9b526b427751c211fb5cf7ff9cd.

2. Update Status.actor to track ClusterController interface to track
   recovery status.
3. Introduce a ServerKnob to define "cluster recovery trace event"
   prefix; for now keeping it as "Master", however, it should allow
   smooth transition to "Cluster" prefix as it seems more appropriate.
 2022-01-06 12:15:51 -08:00
sfc-gh-tclinkenbeard
54eb3b2e91 Disable static assertion in commitBatcher 2022-01-06 12:14:13 -08:00
sfc-gh-tclinkenbeard
648609d764 Make VectorRefPreserializer move constructor noexcept 2021-12-30 22:46:16 -08:00
Aaron Molitor
30b05b469c Revert "Refactor: ClusterController driving cluster-recovery state machine" 
This reverts commit dfe9d184ff5dd66bdbbc5b984688ac3ebb15b901.
 2021-12-24 11:25:51 -08:00
Ata E Husain Bohra
dfe9d184ff Refactor: ClusterController driving cluster-recovery state machine 
At present, cluster recovery process consists of following steps:
1. ClusterController clusterWatchDatabase actor recruits
   master/sequencer process.
2. Sequencer process implements the cluster recovery state machine,
   responsible to recruit all other processes as well restore the
   cluster state.

Patch proposes a scheme where the cluster recovery state machine
is implemented and driven by the ClusterController process instead
of the Sequencer process.

Advantages of the scheme could be:
1. Simplified design where ClusterController recruits "sequencer"
   process like other worker processes compared to current scheme
   where "sequencer" process gets special treatment. In newer scheme
   sequencer is responsible for maintaining/providing
   "committed version" (as expected).
2. ClusterController is responsible for worker processes recruitment,
   the sequencer though orchestrating the recovery state machine, it
   need to reachout to the ClusterController for recruiting worker
   processes etc.

NOTE:
Patch has moved the recovery state machine code from
'sequencer' -> 'cluster-controller' process, however, necessary
updates were done for both functionality as well as performance
improvement reasons.

Next Steps:
Cluster recovery documentation will be updated in near future.
 2021-12-22 14:06:27 -08:00
sfc-gh-tclinkenbeard
0ba77ea79b Fix proxySnapCreate trace typo 2021-11-14 16:12:28 -08:00
Jingyu Zhou
5c258b089f Fix written_tags not propagated to commit proxies 
This is needed so that private mutation tags are propagated from Resolvers to
commit proxies. Otherwise, the master's VV won't be correct and can cause data
corruptions.
 2021-11-11 11:26:13 -08:00
Jingyu Zhou
a8ba9bd021 Fix assertion of resolver returned mutation count 2021-11-09 11:45:53 -08:00
Jingyu Zhou
9f305d67a9 Fix backup mutation tags not counted in version vector 2021-10-22 08:43:20 -07:00
sbodagala
48a0ecd647
Merge pull request #5787 from dlambrig/integrate-PR5700 
version vector / Calculate TPCV on resolvers
 2021-10-20 16:35:44 -04:00
Sreenath Bodagala
06f5401d14 Propagate written tags from proxy to sequencer when ENABLE_VERSION_VECTOR 
is enabled, not when ENABLE_VERSION_VECTOR_TLOG_UNICAST is enabled.
 2021-10-18 16:31:31 +00:00
Dan Lambright
eb814ce070 Remove dead code, check replies match for all resolvers 2021-10-18 10:23:08 -04:00
Dan Lambright
23062b892e Calculate tpcv on resolvers 2021-10-15 16:40:00 -04:00
Jingyu Zhou
dd8840837d Assert private mutation counts are the same from Resolvers 2021-10-15 12:13:20 -04:00
Jingyu Zhou
c6a9ac11a8 fix another conflict fdbserver/ApplyMetadataMutation.cpp 2021-10-15 09:48:33 -04:00
Jingyu Zhou
fc7325e923 Fix force recovery when generating private mutations 
Resolver need to skip transactions after forceRecovery flag is true.
 2021-10-15 09:47:23 -04:00
Jingyu Zhou
34e9932778 Separate \xff keyspace from keyResolvers 
This is needed to make sure all Resolvers process all metadata mutations.
For commit proxy, we add all read/write conflict ranges for metadata mutations
to all resolvers.

We also send lock_aware flag for each transaction to resolvers.
 2021-10-15 09:47:23 -04:00
Jingyu Zhou
fddf16534d Send metadata mutations to all Resolvers 
This is to make sure all Resolvers process them and make the same decision,
as well as maintain the same in-memory states, e.g., keyServers, serverKeys.
 2021-10-15 09:47:23 -04:00
Jingyu Zhou
0dc9c607f4 Add knob PROXY_USE_RESOLVER_PRIVATE_MUTATIONS 
To control proxy to use private mutations from resolvers or not.
 2021-10-15 09:47:23 -04:00
Jingyu Zhou
c1d7b03087 CommitProxy uses Resolver's private mutations 
Note the commit proxy still runs applyMetadataMutations for setting up memory
states, and no longer generates private mutations.
 2021-10-15 09:47:23 -04:00
Jingyu Zhou
90a27bedf5 fix conflict fdbserver/ApplyMetadataMutation.cpp 2021-10-15 09:47:20 -04:00
Jingyu Zhou
9ecdbc39ee Add private mutations to Resolver reply messages 2021-10-15 09:45:57 -04:00
Dan Lambright
b78dd3b9dd Respond to comments 9/24 2021-09-30 11:51:57 -04:00
Dan Lambright
58e1888d8e remove network hop by getting previous commit versions in GetCommitVersionRequest 2021-09-30 11:51:57 -04:00
Sreenath Bodagala
2aa3b44d4e Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype 
- Conflicts:
	fdbserver/LogSystem.h
	fdbserver/LogSystemConfig.h
	fdbserver/TagPartitionedLogSystem.actor.cpp

- Files modified during merge:

modified:   fdbserver/LogSystem.cpp
modified:   fdbserver/LogSystemConfig.cpp
 2021-09-17 19:36:18 +00:00
Xiaoge Su
abf73047ca Enforce std:: specifier rather than using namespace 2021-09-16 19:40:28 -07:00
Dan Lambright
8689e1f106 merge with master 2021-08-30 15:29:08 -04:00
Xiaoge Su
e6b8a519c4 fixup! Make processTransactionStateRequestPart not blocking 2021-08-28 21:10:10 -07:00
Xiaoge Su
74d8ca87d7 Refactor TxnStateRequest handler in commitProxyServerCore 2021-08-27 23:38:34 -07:00
FDB Formatster
69508b980f modify comments to make clang-format and coverage tool play nice 2021-08-27 17:18:00 -07:00
FDB Formatster
2c788c233d apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-08-27 17:07:47 -07:00
Lukas Joswiak
a605fb3852
Merge pull request #5026 from sfc-gh-ljoswiak/fixes/alp6 
Actor sampling
 2021-08-11 13:44:17 -07:00
Steve Atherton
54c7036eaf Move role UIDs for MutationTracking TraceEvents from various inconsistent detail fields into the TraceEvent UID field. 2021-08-10 01:52:36 -07:00
Steve Atherton
e5e8a56b66 Merge branch 'master' of https://github.com/apple/foundationdb into debugging-cleanup 
# Conflicts:
#	fdbbackup/FileDecoder.actor.cpp
 2021-08-09 23:36:57 -07:00
Steve Atherton
faa4154a56 MutationTracking now uses a vector of keys to track. Removed "Mutation" detail from DEBUG_MUTATION() events because they are duplicates of the fields already logged in the returned MutationTracking event, which are now renamed and combined into "Mutation". Removed more toString() calls in TraceEvent detail values. 2021-08-09 23:30:45 -07:00
Sreenath Bodagala
a081c0baa5 Merge remote-tracking branch 'apple-upstream/master' into version-vector-prototype 2021-08-05 22:40:32 +00:00
yao-xiao-github
8609b45354
Add histograms to CommitProxyServer. (#5299) 2021-08-05 09:17:37 -07:00