56 Commits

Author SHA1 Message Date
FDB Formatster
df90cc89de apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-10 10:18:07 -08:00
Andrew Noyes
877997632d Merge branch 'release-6.3' into anoyes/merge-release-6.3-master
Include conflict markers for review purposes
2020-12-04 01:38:07 +00:00
Andrew Noyes
1f541f02be Merge branch 'anoyes/merge-6.2-to-6.3' into anoyes/release-6.3-merge
Merge, leaving conflict markers for now
2020-11-24 16:55:34 +00:00
David Youngworth
d64cf8b9e3 Merge branch 6.3 into master 2020-11-17 11:22:45 -08:00
David Youngworth
d0391db862 Merge branch 'release-6.2' into release-6.3 2020-11-16 10:15:23 -08:00
Xiaoge Su
3a6948c199 Report histogram periodically 2020-11-12 17:04:33 -08:00
Vishesh Yadav
2c56d379b2
Merge pull request #3998 from dongxinEric/misc/attach-dcid-to-process-metrics-when-possible
Attach datacenter id to process, network, machine and memory metrics.
2020-11-06 10:54:23 -08:00
Andrew Noyes
c50e997f60 Make status tests deterministic
This change seems to be incorrect since afaict INetwork::timer isn't
guaranteed to be monotonic. Maybe we can make that guarantee or add an
INetwork::timer_monotonic symbol?
2020-11-05 17:05:34 +00:00
Russell Sears
32c87bbb33 Lightweight, power of two spaced histogram implementation + automatic reporting 2020-11-02 11:13:16 -08:00
Xin Dong
e73d189f88 Attach datacenter id to process, network, machine and memory metrics. 2020-10-30 11:20:40 -07:00
Young Liu
8cc3e4d3c6 Merge release-6.3 into master 2020-10-19 22:51:56 -07:00
Meng Xu
4dff55c4ea Add comment for PriorityStarved metrics
Metrics include:
PriorityStarvedBelowX, PriorityMaxStarvedBelowX and PriorityBusyX
2020-10-16 13:45:03 -07:00
sfc-gh-tclinkenbeard
0ac08f6a9b Replace NULL with nullptr in flow 2020-09-20 11:31:49 -07:00
Evan Tschannen
2f5359fa13 fix: lastRunLoopBusyness did count the currently active time 2020-08-31 09:21:44 -07:00
Evan Tschannen
f6f9aea09e fix: runLoopBusyness was always zero 2020-08-28 09:29:54 -07:00
Evan Tschannen
9e2ee1ed4c fixed lastedZeroBusy; added a knob 2020-08-17 23:16:59 -07:00
Evan Tschannen
c72068d6b5 clients load balance across proxies based on process busyness instead of number of requests 2020-08-12 17:17:21 -07:00
A.J. Beamon
d8690d31cd Merge branch 'master' into per-priority-busy-logging
# Conflicts:
#	flow/Net2.actor.cpp
2020-04-15 08:31:30 -07:00
A.J. Beamon
b1172417f5 Merge branch 'master' into per-priority-busy-logging
# Conflicts:
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/Net2.actor.cpp
2020-04-14 14:22:12 -07:00
A.J. Beamon
e104a2e3a6 Merge commit 'cf01233f28a2c42908656a39f458a4475c1d44a3' into run-loop-busy-profiler
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/NativeAPI.actor.h
#	fdbserver/fdbserver.actor.cpp
#	flow/Net2.actor.cpp
2020-04-14 14:02:24 -07:00
Alex Miller
04498cbc0e Make policy failures be reported as per 1s and not over 5s. 2020-03-13 02:49:06 -07:00
Alex Miller
75e2fffe5a Add a ProcessMetrics.TLSPolicyFailures metric
This reports the number of policy failures over the past 5s interval.
It also is step 1 towards getting this information into status json.
2020-03-13 02:24:37 -07:00
Evan Tschannen
1128666840 added additional logging on the log router 2020-03-05 18:17:06 -08:00
A.J. Beamon
abb75f7eb7 Add logging to indicate the time spent at each priority that exceeds some minimum busyness threshold 2020-02-07 14:34:24 -08:00
A.J. Beamon
182dac7cd5 Convert the slow task profiler into a run loop profiler that also logs when the run loop is 100% busy for a knob-configurable duration. 2020-01-28 12:09:37 -08:00
Evan Tschannen
1f7eb1f738 throttle outgoing tls connections before establishing a network connection
store serverTLSConnectionThrottler map inside of g_network, so that it works properly with simulation
2020-01-12 16:44:30 -08:00
A.J. Beamon
ca01593067 Cap busyness to 1.0 at logging time to cover all cases where it could be measured above. 2020-01-02 15:10:42 -08:00
A.J. Beamon
a093021855 Fix priority time calculation. Track max priority busy rather than seconds squared. 2019-12-17 09:14:54 -08:00
A.J. Beamon
fa6e45a852 Separate AsioReactor sleep and react into two different functions. Track slow tasks and time spent in react, track time spent in launch. Don't track react time at priority 0. 2019-08-28 14:35:48 -07:00
A.J. Beamon
b5d2234a13 Merge branch 'release-6.1' into merge-release-6.1-into-master
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbserver/MoveKeys.actor.cpp
#	flow/FastAlloc.h
#	versions.target
2019-07-30 16:23:42 -07:00
A.J. Beamon
0bfd474e21 Fix huge arena tracking thread-safety issues. Fixes #1846. 2019-07-16 15:22:19 -07:00
A.J. Beamon
38ae352fc5 Fix a merge issue 2019-07-10 09:46:23 -07:00
A.J. Beamon
7f23814841 Track run loop busyness and report it in status. 2019-06-26 14:03:02 -07:00
A.J. Beamon
a8b9d8e34b
Merge pull request #1336 from tclinken/fast-allocate-ptree-nodes
Create 96-byte fast allocator for storage queue PTree nodes
2019-05-17 14:22:46 -07:00
Nikolas Ioannou
c2827f4fa3 Add page cache hit, miss, and eviction stats to SystemMonitor 2019-05-08 15:41:17 +02:00
Evan Tschannen
cd5c9d91fa
Merge pull request #1443 from etschannen/master
Merge 6.1 into master
2019-04-10 17:43:07 -07:00
Evan Tschannen
19d9b8fcd8 don’t log MachineLoadDetails from getSystemStatistics() 2019-04-07 22:55:19 -07:00
mpilman
c008e16c81 Defer formatting in traces to make them cheaper
This is the first part of making `TraceEvent` cheaper. The main idea is
to defer calls to any code that formats string. These are the main
changes:

- TraceEvent::detail now takes a c-string instead of std::string for
  literals. This prevents unnecessary allocations if the trace is not
  going to be printed in the first place (for example for SevDebug).
  Before that `detail` expected a `std::string` as key, which mean that
  any string literal would be copied on each call.
- Templates Traceable and SpecialTraceMetricType. These templates can be
  specialized for any type that needs to be printed. The actual
  formatting will be deferred to after the `enabled` check. This
  provides two benefits: (1) if a TraceEvent is disabled, we don't pay
  for the formatting and (2) TraceEvent can trace types that it doesn't
  know about.
- TraceEvent::enabled will be set in the constructor if the Severity is
  passed. This will make sure that `TraceEvent::init` is not called.
- `TraceEvent::detail` will be inlined. So for disabled TraceEvent
  calls, a call to detail will only introduce a if-branch which is much
  cheaper than a function call.
2019-04-05 13:12:19 -07:00
Trevor Clinkenbeard
dbcf1d717c Added calls to TRACEALLOCATOR(96) and DETAILALLOCATORMEMUSAGE(96) 2019-03-25 13:45:07 -07:00
Evan Tschannen
f73aad863b adding missing logging for 8192 fast allocators 2019-03-20 13:48:45 -07:00
Evan Tschannen
70ac5ffda0 added random logging for huge arenas 2019-03-20 11:20:47 -07:00
Vishesh Yadav
57832e625d net: Support IPv6 #963
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.

- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.

- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.

- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
Trevor Clinkenbeard
78aad255b0 updateProcessStats gets process stats directly
Storage servers no longer parse ProcessMetrics trace lines to get their
own cpuUsage and diskUsage statistics.
2019-02-25 13:45:53 -08:00
Evan Tschannen
684a22a52b Merge branch 'release-6.0'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbclient/BackupContainer.actor.cpp
#	fdbclient/HTTP.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/BackupCorrectness.actor.cpp
#	versions.target
2019-01-09 16:14:46 -08:00
A.J. Beamon
d265517156 Fix: fast allocator would not cleanup memory for a thread if that thread never called getMagazine. This could happen if the first thing the thread did was to release memory.
Added a new metric for the number of threads that hold memory for each size and improve some existing metrics.
Fix: a failed ASSERT would crash if done early in the program lifetime.
2019-01-08 14:36:01 -08:00
Robert Escriva
268093a96d Adjust all includes to be relative to the root.
Remove the use of relative paths.  A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h".  Adjust so that every include references such a header with the
latter form.

Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
A.J. Beamon
afdd992fca Move the computation of total unused allocated memory into FastAlloc.cpp 2018-06-20 14:33:59 -07:00
A.J. Beamon
5e81f4ac7e Track unused allocated memory in ProcessMetrics and report it in status. 2018-06-20 10:10:51 -07:00
A.J. Beamon
99c9958db7 Some more trace event normalization 2018-06-08 13:57:00 -07:00
A.J. Beamon
e5488419cc Attempt to normalize trace events:
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.

Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.

This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00