122 Commits

Author SHA1 Message Date
A.J. Beamon
b5d2234a13 Merge branch 'release-6.1' into merge-release-6.1-into-master
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbbackup/backup.actor.cpp
#	fdbserver/MoveKeys.actor.cpp
#	flow/FastAlloc.h
#	versions.target
2019-07-30 16:23:42 -07:00
A.J. Beamon
e98cee016d Fix unsafe usage of now() function from multiple threads in trace logging. 2019-07-22 22:31:38 -07:00
A.J. Beamon
d981de18e4 Restrict huge arena sampling to the network thread. Revert removal of thread_local definitions. 2019-07-17 16:23:17 -07:00
A.J. Beamon
d5051b08dd Make trace event field lengths (and total event sizes) default knobified and configurable. Add a transaction option to control the field length of transaction debug logging. Make the program start command line field less likely to be truncated. 2019-07-12 16:12:35 -07:00
Alex Miller
7a500cd37f A giant translation of TaskFooPriority -> TaskPriority::Foo
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
A.J. Beamon
f417e60264 Merge branch 'merge-release-6.1-into-master' into thread-safe-random-number-generation
# Conflicts:
#	fdbserver/QuietDatabase.actor.cpp
2019-05-23 09:52:00 -07:00
A.J. Beamon
d29c7e4c9b Merge branch 'release-6.1' into merge-release-6.1-into-master
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbserver/QuietDatabase.actor.cpp
#	versions.target
2019-05-23 09:28:45 -07:00
A.J. Beamon
603721e125 Merge branch 'master' into thread-safe-random-number-generation
# Conflicts:
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/AsyncFileCached.actor.h
#	fdbrpc/genericactors.actor.cpp
#	fdbrpc/sim2.actor.cpp
#	fdbserver/DiskQueue.actor.cpp
#	fdbserver/workloads/BulkSetup.actor.h
#	flow/ActorCollection.actor.cpp
#	flow/Net2.actor.cpp
#	flow/Trace.cpp
#	flow/flow.cpp
2019-05-23 08:35:47 -07:00
A.J. Beamon
d3c1f9afbd Suppress 'TraceEventFieldNotFound' event. Don't suppress simultaneous FDBLibTLSVerifyFailure events (this requires including flow.h and removing the copied version of Reference/ReferenceCounted in FDBLibTLS). 2019-05-20 15:12:28 -07:00
A.J. Beamon
ab4618c3f9 Don't access TraceEvent::eventCounts outside the network thread. Make net2liveness an atomic counter. 2019-05-10 12:35:11 -10:00
A.J. Beamon
5f55f3f613 Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used. 2019-05-10 14:01:52 -07:00
Evan Tschannen
22499666d0 Merge branch 'release-6.1'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbserver/LogRouter.actor.cpp
#	flow/Trace.cpp
#	versions.target
2019-05-08 18:19:35 -07:00
Evan Tschannen
aefd68e1e7 The memory tracking trace events can crash if the memory that is being allocated is coming from a trace event itself. Specifically TDMetrics within trace events uses fast allocated memory. Trace events are supposed to be short lived, so this commits adds a global count of outstanding trace events, and disables memory tracking when a trace event exists. 2019-05-06 17:41:32 -07:00
Andrew Noyes
6c7fd1593f Avoid potential index-out-of-bounds 2019-04-15 15:20:09 -07:00
Andrew Noyes
dced9232d0 ASSERT_WE_THINK trailing byte is '\0' 2019-04-15 09:43:26 -07:00
mpilman
d01cbf3455 Addressed code review comments 2019-04-05 13:12:20 -07:00
mpilman
8120a33bda Fix >-direction when initializing TraceEvent 2019-04-05 13:12:19 -07:00
mpilman
c008e16c81 Defer formatting in traces to make them cheaper
This is the first part of making `TraceEvent` cheaper. The main idea is
to defer calls to any code that formats string. These are the main
changes:

- TraceEvent::detail now takes a c-string instead of std::string for
  literals. This prevents unnecessary allocations if the trace is not
  going to be printed in the first place (for example for SevDebug).
  Before that `detail` expected a `std::string` as key, which mean that
  any string literal would be copied on each call.
- Templates Traceable and SpecialTraceMetricType. These templates can be
  specialized for any type that needs to be printed. The actual
  formatting will be deferred to after the `enabled` check. This
  provides two benefits: (1) if a TraceEvent is disabled, we don't pay
  for the formatting and (2) TraceEvent can trace types that it doesn't
  know about.
- TraceEvent::enabled will be set in the constructor if the Severity is
  passed. This will make sure that `TraceEvent::init` is not called.
- `TraceEvent::detail` will be inlined. So for disabled TraceEvent
  calls, a call to detail will only introduce a if-branch which is much
  cheaper than a function call.
2019-04-05 13:12:19 -07:00
Jingyu Zhou
38c6681349 Fix some signed and unsigned mismatch warnings. 2019-03-26 14:54:11 -07:00
Vishesh Yadav
c37291a366 fix: Use '_' instead of ':' in IPv6 tracefile names
':' is not acceptable on Windows. Reason to choose '_' instead of '.'
is to differentiate b/w IPv4 and IPv6 easily, and also '..1' in
filename looks weirder than '__1', which would happen with shortened
IPv6 addresses. And non-shortened IPv6 addreses are just lots of 0s.
2019-03-20 14:00:33 -07:00
Vishesh Yadav
592e224155 net: add/use formatIpPort to format IP:PORT pairs #963 2019-03-04 14:12:45 -08:00
Vishesh Yadav
25daabdc02 net: TraceEvent and toIPVectorString for new IPAddress structure #963 2019-03-04 14:12:45 -08:00
Vishesh Yadav
57832e625d net: Support IPv6 #963
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.

- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.

- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.

- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
A.J. Beamon
655c9d82c7 Various cleanup from review 2019-03-01 14:06:47 -08:00
A.J. Beamon
eb629d87a5 Add information about batch ratekeeper to status. Make it possible to track latencies in the ReadWrite workload for concurrently run instances separately. 2019-02-28 09:53:16 -08:00
Andrew Noyes
e055fdab03 Better diagnostics for unrecognized trace format 2019-01-29 13:08:47 -08:00
Andrew Noyes
3f399dd9be Indent within anonymous namespace 2019-01-03 09:21:09 -08:00
Andrew Noyes
80dd1efaf6 Assign to g_traceLog.formatter 2019-01-02 16:42:00 -08:00
anoyes
b8df5acc15 Add --trace_format flag to fdbserver 2018-12-20 15:02:01 -08:00
anoyes
33a4aac249 Fix build 2018-12-20 11:55:59 -08:00
Markus Pilman
cdc7e83993
Update flow/Trace.cpp
Co-Authored-By: alexmiller-apple <35046903+alexmiller-apple@users.noreply.github.com>
2018-12-14 11:30:40 -08:00
Markus Pilman
40890e9dbe Implemented a json log formatter 2018-12-13 21:46:02 -08:00
Robert Escriva
268093a96d Adjust all includes to be relative to the root.
Remove the use of relative paths.  A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h".  Adjust so that every include references such a header with the
latter form.

Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
A.J. Beamon
9e79f9ec59 Cleanup from review 2018-09-05 15:53:12 -07:00
A.J. Beamon
2de0b5d6d7 Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations. 2018-09-05 15:06:14 -07:00
A.J. Beamon
168dce94cb Remove some trace event suppressions that were happening off the network thread. Downgrade some trace events related to trace logging problems from SevError to SevWarnAlways. 2018-08-14 09:00:43 -07:00
Evan Tschannen
1c29275672 call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details. 2018-08-01 14:30:57 -07:00
Evan Tschannen
5054c194e2 Some trace events are logged before FLOW_KNOBS are initialized 2018-07-01 14:30:37 -04:00
A.J. Beamon
a680837ee4 The Machine field was missing in early trace events. The logGroup field was not being properly set. 2018-06-28 15:28:58 -07:00
A.J. Beamon
1ff42e078f Add missing include for Windows, remove throw from TraceEvent destructor. 2018-06-28 14:59:23 -07:00
A.J. Beamon
cbc840ad0a Move the spammy trace event backstop from the destructor to the constructor. This allows us to avoid doing needless work on a trace event that is going to be throttled. 2018-06-27 15:51:30 -07:00
A.J. Beamon
fe956bc35a Address review comments 2018-06-26 14:37:21 -07:00
A.J. Beamon
f965954122 Merge commit '82be52205b95464e355c449fdf3e7d483fa06677' into trace-log-refactor
# Conflicts:
#	fdbserver/Status.actor.cpp
#	fdbserver/workloads/DDMetrics.actor.cpp
#	flow/Trace.cpp
2018-06-08 16:22:22 -07:00
A.J. Beamon
06ccd9a500 Allow trace event type names to end with an underscore. 2018-06-08 15:49:31 -07:00
A.J. Beamon
1fdfe20908 Relax the rules on trace event Types a bit by allowing multiple underscores, as well as starting with an underscore and consecutive underscores. 2018-06-08 15:40:29 -07:00
A.J. Beamon
0ca51989bb Merge branch 'master' into trace-log-refactor
# Conflicts:
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/Status.actor.cpp
#	flow/Trace.cpp
2018-06-08 13:24:30 -07:00
A.J. Beamon
e5488419cc Attempt to normalize trace events:
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.

Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.

This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
A.J. Beamon
3ea5fc72f0 Rename trace file logGroup attribute to LogGroup 2018-05-31 15:34:17 -07:00
A.J. Beamon
f1dea3bed9 Rework the formatter a bit and extract out the writing logic so that it could be pluggable. 2018-05-31 13:27:35 -07:00
A.J. Beamon
f7735d000f Fix linking issue in flow bindings by using a different variant of printable(). 2018-05-31 11:16:01 -07:00