2452 Commits

Author SHA1 Message Date
Jingyu Zhou
8e9aaa767e
Merge pull request #1860 from alexmiller-apple/comma-operator
Remove `operator , (vector<T>, T)` as an append operator.
2019-07-19 12:53:32 -07:00
A.J. Beamon
f6183df8b9
Merge pull request #1852 from vishesh/task/issue-1840-non-blocking-exclusion
fdbcli: Add `no_wait` option in `exclude` command to avoid blocking
2019-07-19 12:49:29 -07:00
Jingyu Zhou
63e37aebaf Reorder include files. 2019-07-19 11:24:26 -07:00
Jingyu Zhou
d8fb1ea2d3 Log large transactions at proxy
This can help debugging where large transactions are coming from.
2019-07-19 11:10:48 -07:00
A.J. Beamon
bc5c65e5ab
Merge pull request #1756 from jzhou77/db-option
Add transaction getApproximateSize() API
2019-07-19 08:33:24 -07:00
Alex Miller
9863ace96c Replace usages with intialization lists.
But C++ needs a bit of help to inference though the templates.
2019-07-18 22:27:36 -07:00
Meng Xu
64bee63dbc Resolve two review comments
1) No need to check server with only one team when teamRemover finds
a server team or machine team to remove

2) Fix optimalTeamCount counting in teamTracker
2019-07-18 18:46:31 -07:00
Vishesh Yadav
d9a8657096 fdbcli: Add no_wait option in exclude command to avoid blocking
RESOLVES #1840
2019-07-18 13:07:31 -07:00
Balachandar Namasivayam
14e54f44b3 Address review comments. 2019-07-18 12:32:35 -07:00
Evan Tschannen
94c66f8d58
Merge pull request #1738 from bnamasivayam/consistency-check-disable
Disable/Re-enable consistency check through a database key.
2019-07-18 10:56:02 -07:00
mpilman
1ac2d01b03 Merge remote-tracking branch 'upstream/master' into flatbuffers-fixes2 2019-07-18 09:50:08 -07:00
Balachandar Namasivayam
ecb3de3b49 Fixed space issue. 2019-07-17 18:10:05 -07:00
Balachandar Namasivayam
406bcebdc4 Ratekeeper to throttle tpsLimit to 1 if it is not able to fetch storage server list for some configurable amount of time. 2019-07-17 18:08:17 -07:00
Balachandar Namasivayam
e08c25ffd8 Style fix. 2019-07-17 17:31:50 -07:00
Alex Miller
55258709a0 Remove an ASSERT from testing and now inaccurate comment. 2019-07-17 01:30:01 -07:00
Alex Miller
e9684a1f63 Fix issues configuring from sharded txs tag to not
Which is an intermingling of what should be two commits:

1. Rely on TLogVersion instead of txsTags==0

2. Copy and index sharded txsTags between KCV and RV as txsTag when
configuring log_version 4->3.
2019-07-17 01:25:09 -07:00
Alex Miller
95487861be Make sharded txsTag gated on TLogVersion::V4.
To allow a potential 6.2 -> 6.1 rollback.
2019-07-16 19:09:53 -07:00
Alex Miller
9396eedd11 Const some random functions that are trivially const.
For code hygiene reasons only.
2019-07-16 19:09:09 -07:00
Meng Xu
6df93173ca TC:Lower priority for removing redundant teams 2019-07-16 18:02:36 -07:00
Jingyu Zhou
690ffb91b0
Merge pull request #1850 from alexmiller-apple/apply-metadata-mutations
Move code in ApplyMetadataMutation from .h to .cpp
2019-07-16 16:59:50 -07:00
Alex Miller
812ce37bcd Remove buggify and unneeded safeguards.
The buggify was actually incorrect and broke an invariant, which I then
fixed on the other side, but this work was actually unneeded in total.

The real issue being fixed was returnIfBlock not sending an error, as
well as the other error cases.
2019-07-16 15:58:02 -07:00
Alex Miller
a79c5d8200 Move code in ApplyMetadataMutation from .h to .cpp
Not only can we now compile this giant code block in parallel with the
rest of the code, we also no longer compile this giant function twice!
(Once for the master, once for the proxy.)
2019-07-16 15:00:34 -07:00
mpilman
d5caf0c1b4 Merge branch 'flatbuffers-fixes2' of github.com:mpilman/foundationdb into flatbuffers-fixes2 2019-07-16 14:47:40 -07:00
Meng Xu
80ed39c189 QuietDB:Disable check for too many teams
Because team remover does not remove a team if it causes 0 team per server.
So we currently disable the check until we have a better strategy to enforce the
desired number of teams.

This will not cause much problem in real situation, while having 0 team on a server
will make the server unable to host data, which is bad.
2019-07-16 12:38:55 -07:00
Meng Xu
915732ce24 TeamRemover:Reset the removed team counter after removement 2019-07-16 11:17:51 -07:00
Jingyu Zhou
d87a133cfc Make takeMoveKeysLock() update move key owner lock
This allows a new data distributor to update the owner lock so that older
data distributor can't hang on to the lock. This fixes #1821
2019-07-16 11:06:20 -07:00
Meng Xu
20f067e794 Merge with master:Resolve conflict with PR#1797 2019-07-16 10:52:28 -07:00
Meng Xu
f5f250d117
Merge pull request #1797 from ajbeamon/trace-event-rename
Rename event details that use the suffix "Number" to indicate a count…
2019-07-15 19:22:07 -07:00
Meng Xu
243504b125 DD:Clang format changes 2019-07-15 18:40:14 -07:00
Meng Xu
94e9b8a3b4 Do not remove a team whose min team number is less than target
If the minimum number of teams of servers in a team is less than the
target value (desired_team_number_per_server * (teamSize + 1) / 2),
the team remover should not remove it. Otherwise, DD will oscillate in
building more teams and removing redundant teams.

Do not do consistency check for three_data_hall mode because when
machines are not evenly distributed across data halls, we will
need to build more teams than the total desired number to make sure
the number of teams per server is no less than the target value.
2019-07-15 18:30:13 -07:00
Meng Xu
cafe9b9412 TC:Target team num per server is desired number
Do not overbuild teams because we may oscillate between building more teams and
removing the redundant teams. The oscillation happens when the machines are not
evenly distributed across availability zones.
For example, in three_data_hall mode, we have 1 machine in 1 data hall for 2 data halls.
We have 3 machines in the 3rd data hall. To build enough (and more teams) for servers
in the 3rd data hall, we will overbuild teams. However,
the teamRemover will remove those newly teams.
2019-07-15 17:32:51 -07:00
Alex Miller
4cc60dc9b8 Merge remote-tracking branch 'upstream/master' into peek-cursor-timeout-bug 2019-07-15 17:05:39 -07:00
Alex Miller
32af11286b Copy same changes to OldTLogServer_6_0. 2019-07-15 16:43:43 -07:00
Alex Miller
2cbc05fc72 Address more issues that cause peek cursors to time out.
There were error cases that would cause a peek to terminate early or be
cancelled without sending anything to the next peek in line.  We would
thus end up with the first peek in a sequence waiting on its future, and
nothing that exists that would send to that future.
2019-07-15 16:03:37 -07:00
Alec Grieser
a89d09f1a6
Merge pull request #1844 from ajbeamon/configurable-trace-field-length
Make trace event field lengths default knobified and individually configurable.
2019-07-15 15:53:22 -07:00
A.J. Beamon
2cd05e9ac9
Merge pull request #1712 from tclinken/add-local-rk-to-status
Track the local ratekeeper rate in status
2019-07-15 15:17:11 -07:00
mpilman
54416f46fd Pass type as param to VectorRef instead of bool 2019-07-15 15:08:49 -07:00
Meng Xu
415622f465 MachineTeamRemover:Change to remove MT with most teams
Change to remove machine team with most machine teams, using the same
logic as the serverTeamRemover.

The featue is guarded by TR_FLAG_REMOVE_MT_WITH_MOST_TEAMS knob.
2019-07-15 14:29:49 -07:00
Balachandar Namasivayam
9169232fa9 Add the new messages to Schema. 2019-07-15 13:47:27 -07:00
Alex Miller
c8e94e601a
Merge pull request #1729 from etschannen/feature-fast-txs-recovery
Improve the recovery speed of the txnStateStore
2019-07-15 13:27:41 -07:00
Balachandar Namasivayam
4a99bd2961 Addressed review comments. 2019-07-15 12:33:18 -07:00
Jon Fu
27c9414d66 changed trace detail to uppercase to pass validation 2019-07-15 11:15:13 -07:00
Trevor Clinkenbeard
e1541778ab Added readsRejected counter to storage server 2019-07-15 10:53:19 -07:00
Jon Fu
4b0fdabae5 mark test file as IGNORE and comment out dead placeholder code 2019-07-15 09:45:16 -07:00
Evan Tschannen
d4c332e2cf update lastCommitTime when doing GRV’s without causal read risky 2019-07-14 14:46:00 -07:00
Meng Xu
5c5e883745 TC:Keep building until each server and machine has at least the expected number of teams 2019-07-12 19:16:18 -07:00
Evan Tschannen
db5b4a6331 avoid going to unlimited immediately after going below the durabilityLagTargetVersion 2019-07-12 18:50:56 -07:00
Evan Tschannen
6e34e16699 durable version needs more smoothing because it will be updated in bursts 2019-07-12 18:50:56 -07:00
Evan Tschannen
b2b2e25324 the durabilityLagLimit needs to be tracked separately for batch priority and normal priority 2019-07-12 18:50:56 -07:00
Evan Tschannen
fef58e13a4 adding logging for durability lag in ratekeeper 2019-07-12 18:50:56 -07:00