38 Commits

Author SHA1 Message Date
Xiaoge Su
3a6948c199 Report histogram periodically 2020-11-12 17:04:33 -08:00
Vishesh Yadav
2c56d379b2
Merge pull request #3998 from dongxinEric/misc/attach-dcid-to-process-metrics-when-possible
Attach datacenter id to process, network, machine and memory metrics.
2020-11-06 10:54:23 -08:00
Russell Sears
32c87bbb33 Lightweight, power of two spaced histogram implementation + automatic reporting 2020-11-02 11:13:16 -08:00
Xin Dong
e73d189f88 Attach datacenter id to process, network, machine and memory metrics. 2020-10-30 11:20:40 -07:00
Alex Miller
04498cbc0e Make policy failures be reported as per 1s and not over 5s. 2020-03-13 02:49:06 -07:00
Alex Miller
75e2fffe5a Add a ProcessMetrics.TLSPolicyFailures metric
This reports the number of policy failures over the past 5s interval.
It also is step 1 towards getting this information into status json.
2020-03-13 02:24:37 -07:00
Evan Tschannen
1128666840 added additional logging on the log router 2020-03-05 18:17:06 -08:00
Evan Tschannen
1f7eb1f738 throttle outgoing tls connections before establishing a network connection
store serverTLSConnectionThrottler map inside of g_network, so that it works properly with simulation
2020-01-12 16:44:30 -08:00
A.J. Beamon
ca01593067 Cap busyness to 1.0 at logging time to cover all cases where it could be measured above. 2020-01-02 15:10:42 -08:00
A.J. Beamon
a093021855 Fix priority time calculation. Track max priority busy rather than seconds squared. 2019-12-17 09:14:54 -08:00
A.J. Beamon
fa6e45a852 Separate AsioReactor sleep and react into two different functions. Track slow tasks and time spent in react, track time spent in launch. Don't track react time at priority 0. 2019-08-28 14:35:48 -07:00
A.J. Beamon
b5d2234a13 Merge branch 'release-6.1' into merge-release-6.1-into-master
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbserver/MoveKeys.actor.cpp
#	flow/FastAlloc.h
#	versions.target
2019-07-30 16:23:42 -07:00
A.J. Beamon
0bfd474e21 Fix huge arena tracking thread-safety issues. Fixes #1846. 2019-07-16 15:22:19 -07:00
A.J. Beamon
38ae352fc5 Fix a merge issue 2019-07-10 09:46:23 -07:00
A.J. Beamon
7f23814841 Track run loop busyness and report it in status. 2019-06-26 14:03:02 -07:00
A.J. Beamon
a8b9d8e34b
Merge pull request #1336 from tclinken/fast-allocate-ptree-nodes
Create 96-byte fast allocator for storage queue PTree nodes
2019-05-17 14:22:46 -07:00
Nikolas Ioannou
c2827f4fa3 Add page cache hit, miss, and eviction stats to SystemMonitor 2019-05-08 15:41:17 +02:00
Evan Tschannen
cd5c9d91fa
Merge pull request #1443 from etschannen/master
Merge 6.1 into master
2019-04-10 17:43:07 -07:00
Evan Tschannen
19d9b8fcd8 don’t log MachineLoadDetails from getSystemStatistics() 2019-04-07 22:55:19 -07:00
mpilman
c008e16c81 Defer formatting in traces to make them cheaper
This is the first part of making `TraceEvent` cheaper. The main idea is
to defer calls to any code that formats string. These are the main
changes:

- TraceEvent::detail now takes a c-string instead of std::string for
  literals. This prevents unnecessary allocations if the trace is not
  going to be printed in the first place (for example for SevDebug).
  Before that `detail` expected a `std::string` as key, which mean that
  any string literal would be copied on each call.
- Templates Traceable and SpecialTraceMetricType. These templates can be
  specialized for any type that needs to be printed. The actual
  formatting will be deferred to after the `enabled` check. This
  provides two benefits: (1) if a TraceEvent is disabled, we don't pay
  for the formatting and (2) TraceEvent can trace types that it doesn't
  know about.
- TraceEvent::enabled will be set in the constructor if the Severity is
  passed. This will make sure that `TraceEvent::init` is not called.
- `TraceEvent::detail` will be inlined. So for disabled TraceEvent
  calls, a call to detail will only introduce a if-branch which is much
  cheaper than a function call.
2019-04-05 13:12:19 -07:00
Trevor Clinkenbeard
dbcf1d717c Added calls to TRACEALLOCATOR(96) and DETAILALLOCATORMEMUSAGE(96) 2019-03-25 13:45:07 -07:00
Evan Tschannen
f73aad863b adding missing logging for 8192 fast allocators 2019-03-20 13:48:45 -07:00
Evan Tschannen
70ac5ffda0 added random logging for huge arenas 2019-03-20 11:20:47 -07:00
Vishesh Yadav
57832e625d net: Support IPv6 #963
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.

- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.

- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.

- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
Trevor Clinkenbeard
78aad255b0 updateProcessStats gets process stats directly
Storage servers no longer parse ProcessMetrics trace lines to get their
own cpuUsage and diskUsage statistics.
2019-02-25 13:45:53 -08:00
Evan Tschannen
684a22a52b Merge branch 'release-6.0'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbclient/BackupContainer.actor.cpp
#	fdbclient/HTTP.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/BackupCorrectness.actor.cpp
#	versions.target
2019-01-09 16:14:46 -08:00
A.J. Beamon
d265517156 Fix: fast allocator would not cleanup memory for a thread if that thread never called getMagazine. This could happen if the first thing the thread did was to release memory.
Added a new metric for the number of threads that hold memory for each size and improve some existing metrics.
Fix: a failed ASSERT would crash if done early in the program lifetime.
2019-01-08 14:36:01 -08:00
Robert Escriva
268093a96d Adjust all includes to be relative to the root.
Remove the use of relative paths.  A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h".  Adjust so that every include references such a header with the
latter form.

Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
A.J. Beamon
afdd992fca Move the computation of total unused allocated memory into FastAlloc.cpp 2018-06-20 14:33:59 -07:00
A.J. Beamon
5e81f4ac7e Track unused allocated memory in ProcessMetrics and report it in status. 2018-06-20 10:10:51 -07:00
A.J. Beamon
99c9958db7 Some more trace event normalization 2018-06-08 13:57:00 -07:00
A.J. Beamon
e5488419cc Attempt to normalize trace events:
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.

Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.

This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
Balachandar Namasivayam
336a54faef Split ProcessMetrics into ProcessMetrics, NetworkMetrics and MemoryMetrics 2018-03-06 15:52:03 -08:00
Alec Grieser
e1162e9238 Merge remote-tracking branch 'upstream/release-5.1' 2018-02-22 11:16:12 -08:00
Alec Grieser
0bae9880f1 remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py 2018-02-21 10:25:11 -08:00
Balachandar Namasivayam
2eeb714c4b Trace memory usage of individual FastAllocator of different sizes as part of ProcessMetrics. 2018-02-16 13:30:00 -08:00
A.J. Beamon
b20ae356b1 Alloc instrumentation backtraces use format_backtrace; Magnesium detects backtraces from binaries besides fdbserver. 2017-10-12 08:39:13 -07:00
FDB Dev Team
a674cb4ef4 Initial repository commit 2017-05-25 13:48:44 -07:00