Meng Xu
27a21e23bd
Add number comment to limitReason_t entries
2020-07-15 10:57:50 -07:00
A.J. Beamon
11b136c745
Various fixes to tag throttling:
...
* Master proxy reports transaction counts to ratekeeper for throttled tags only
* The ramp up behavior at the end of an auto-throttle was broken
* Fixed some issues with computing the initial transaction rate for auto-throttles
2020-06-30 16:24:41 -07:00
Evan Tschannen
ced65cd30b
finished explicitly versioning everything stored in the database
2020-05-22 17:14:21 -07:00
A.J. Beamon
cc4874918a
Merge branch 'release-6.3' into tag-throttling-by-priority
...
# Conflicts:
# fdbserver/Ratekeeper.actor.cpp
2020-05-20 14:26:35 -07:00
A.J. Beamon
14b23c146f
Support throttling and unthrottling tags by priority and their auto/manual state in fdbcli.
2020-05-15 12:47:55 -07:00
A.J. Beamon
d3f465fd56
Merge pull request #3102 from mpilman/features/trace-roles
...
Emit traces regularly about role assignment
2020-05-15 08:12:25 -07:00
A.J. Beamon
3ee4912312
Merge pull request #3152 from ajbeamon/tag-throttling-status-improvements
...
Add and fix tag throttling status fields
2020-05-14 16:08:05 -07:00
Markus Pilman
2cdcab5aa7
address review comments
2020-05-14 14:54:38 -07:00
Markus Pilman
5230668a76
Pass Ratekeeper ID to all RK traces
2020-05-14 14:17:43 -07:00
Markus Pilman
c2bc75516f
Merge branch 'release-6.3' of github.com:apple/foundationdb into features/trace-roles
2020-05-14 10:34:53 -07:00
A.J. Beamon
634c988059
Tag throttles reacted poorly to the rate being set to max
2020-05-12 18:09:43 -07:00
A.J. Beamon
acf1244317
Fix: ratekeeper was logging the auto throttle count for the manual throttle count
2020-05-12 14:10:40 -07:00
Markus Pilman
5f9b127e56
Emit traces regularly about role assignment
...
We are currently emitting Role transition traces when a role starts and
when it ends. While this is useful for debugging, it doesn't work well
with tools that inject data and might potentially miss some trace lines.
We do decorate each trace lines with the roles assigned to that
particular process, however, this is not sufficient for tools that can
make use of the UID -> Role mapping
2020-05-08 16:27:57 -07:00
A.J. Beamon
fbf436f45f
Various cleanup and knob adjustments.
2020-05-07 09:15:33 -07:00
A.J. Beamon
35d382811f
Fix compilation error
2020-05-07 08:59:47 -07:00
A.J. Beamon
66b4920fc3
Fix off-by-one error in auto-throttle limit. Allow updating existing auto-throttle when the limit is reached.
2020-05-05 09:58:00 -07:00
A.J. Beamon
bb3d4b6b89
Add a bunch of TEST macros and some other little things
2020-05-04 10:11:36 -07:00
A.J. Beamon
31cef6075a
Do the auto-throttle ramp up in a better place. Only commit manual throttle limit once. Add some asserts.
2020-05-03 19:15:29 -07:00
A.J. Beamon
decf3e82b0
Fix various bugs and make sure to cleanup throttles from the database when they expire
2020-05-01 21:36:28 -07:00
A.J. Beamon
3a7a026aae
Fix a couple bugs
2020-04-30 22:24:17 -07:00
A.J. Beamon
b80225dde0
Initial support for ramping load back up. Fix some logging. Update auto-throttles less frequently.
2020-04-28 15:50:45 -07:00
A.J. Beamon
0ed70accfa
Reorganization of throttle storage in ratekeeper to support various auto-throttling related actions
2020-04-28 14:30:37 -07:00
A.J. Beamon
a65e97209a
Fix bug in flag management. Pass priority into ratekeeper's updateRate and use it when setting autothrottles.
2020-04-27 11:34:12 -07:00
A.J. Beamon
239876351b
Add some initial auto-throttling. Move the definition of the priority enum to a more global place and use it for all transaction priorites (except in ClientLogEvents, because of serialization incompatibilites).
2020-04-24 11:31:16 -07:00
A.J. Beamon
7343c1b333
Logging for throttle changes was moved
2020-04-23 20:51:53 -07:00
A.J. Beamon
35c18ac60a
Improvements to expiration. Encode throttles with auto/manual and priority in the key to support throttling the same tag with different values in these parameters.
2020-04-23 20:50:40 -07:00
A.J. Beamon
18f860d9d8
Minor cleanup around expiration on ratekeeper.
2020-04-23 10:42:16 -07:00
A.J. Beamon
d2504c08c3
Send throttles at all priorities from RK->MP
2020-04-22 13:47:16 -07:00
A.J. Beamon
434704fbd9
Various bug fixes
2020-04-22 12:28:51 -07:00
A.J. Beamon
f1dd0ee298
Protect against a ratekeeper starting up with a clock set in the past (compared to old ratekeeper) extending the duration of throttles excessively.
2020-04-21 16:35:25 -07:00
A.J. Beamon
d5fb4d26fe
Send tag throttle updates from ratekeeper to proxy only when they change
2020-04-21 16:33:56 -07:00
A.J. Beamon
b2c14611f3
Ratekeeper maintains its throttle data a bit differently, and now it aggregates the effects of multiple priorities before sending results to the proxy
2020-04-21 11:58:59 -07:00
A.J. Beamon
dfec896438
Enforce a throttle limit. Don't count transaction tags on RK if the proxy has updated us in a while.
2020-04-17 11:48:02 -07:00
A.J. Beamon
6619a1a36a
Rename transaction tag map.
2020-04-17 09:06:45 -07:00
A.J. Beamon
2b66dcd24a
Some more refactoring. Reduce what is sent from RK->MP->clients
2020-04-17 08:07:01 -07:00
A.J. Beamon
0fba8c47be
Checkpoint: Ratekeeper sets absolute limits for tag throttles and enforces them by distributing requests to proxies, who distribute them to clients.
...
A few refactorings.
2020-04-16 14:43:22 -07:00
A.J. Beamon
7f3fa00897
Reorganize a bit of code.
2020-04-10 13:53:23 -07:00
A.J. Beamon
29b2c2f3aa
Add hash to StringRef. Use unordered maps for storing tags. Create some helpful typedefs.
2020-04-10 12:54:59 -07:00
A.J. Beamon
55a0d00ad4
Encoding of tags in the database now supports multiple tags per throttle. Remove throttle prefix search.
2020-04-10 10:12:26 -07:00
A.J. Beamon
ebeca10bce
Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients.
2020-04-09 16:55:56 -07:00
A.J. Beamon
a1d8623e5f
Various changes to the throttling scheme, including most notably that clients now enforce the throttles with info they receive from the proxy.
2020-04-07 16:28:09 -07:00
A.J. Beamon
2336f073ad
Checkpointing a bunch of work on throttles. Rudimentary implementation of auto-throttling. Support for manual throttling via fdbcli. Throttles are stored in the system keyspace.
2020-04-03 15:24:14 -07:00
Evan Tschannen
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
Evan Tschannen
1128666840
added additional logging on the log router
2020-03-05 18:17:06 -08:00
Evan Tschannen
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
Evan Tschannen
08914a2acd
Once available space ratio falls below 0.3 avoid moving data to teams with less free space than the median team
2020-02-21 15:14:32 -08:00
Evan Tschannen
819c55556c
More aggressively attempt to find teams that do not have low disk space
2020-02-20 16:47:50 -08:00
Evan Tschannen
bf7d7e2f1e
Merge pull request #2499 from ajbeamon/ratekeeper-durable-version-smoother-fix
...
Fix inaccurate limiting durability lag
2020-02-04 13:04:58 -08:00
Evan Tschannen
3f9d9d8b84
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# cmake/FlowCommands.cmake
# documentation/sphinx/source/release-notes.rst
# fdbclient/StorageServerInterface.h
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# flow/Knobs.h
# flow/Platform.cpp
# versions.target
2020-01-16 18:37:47 -08:00
A.J. Beamon
9668c4471f
Clamp infinite limit in ratekeeper
2020-01-14 15:45:24 -08:00
