Commit Graph

62 Commits

Author SHA1 Message Date
A.J. Beamon ab4618c3f9 Don't access TraceEvent::eventCounts outside the network thread. Make net2liveness an atomic counter. 2019-05-10 12:35:11 -10:00
Evan Tschannen 22499666d0 Merge branch 'release-6.1'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbserver/LogRouter.actor.cpp
#	flow/Trace.cpp
#	versions.target
2019-05-08 18:19:35 -07:00
Evan Tschannen aefd68e1e7 The memory tracking trace events can crash if the memory that is being allocated is coming from a trace event itself. Specifically TDMetrics within trace events uses fast allocated memory. Trace events are supposed to be short lived, so this commits adds a global count of outstanding trace events, and disables memory tracking when a trace event exists. 2019-05-06 17:41:32 -07:00
Andrew Noyes 6c7fd1593f Avoid potential index-out-of-bounds 2019-04-15 15:20:09 -07:00
Andrew Noyes dced9232d0 ASSERT_WE_THINK trailing byte is '\0' 2019-04-15 09:43:26 -07:00
mpilman d01cbf3455 Addressed code review comments 2019-04-05 13:12:20 -07:00
mpilman 8120a33bda Fix `>`-direction when initializing TraceEvent 2019-04-05 13:12:19 -07:00
mpilman c008e16c81 Defer formatting in traces to make them cheaper
This is the first part of making `TraceEvent` cheaper. The main idea is
to defer calls to any code that formats string. These are the main
changes:

- TraceEvent::detail now takes a c-string instead of std::string for
  literals. This prevents unnecessary allocations if the trace is not
  going to be printed in the first place (for example for SevDebug).
  Before that `detail` expected a `std::string` as key, which mean that
  any string literal would be copied on each call.
- Templates Traceable and SpecialTraceMetricType. These templates can be
  specialized for any type that needs to be printed. The actual
  formatting will be deferred to after the `enabled` check. This
  provides two benefits: (1) if a TraceEvent is disabled, we don't pay
  for the formatting and (2) TraceEvent can trace types that it doesn't
  know about.
- TraceEvent::enabled will be set in the constructor if the Severity is
  passed. This will make sure that `TraceEvent::init` is not called.
- `TraceEvent::detail` will be inlined. So for disabled TraceEvent
  calls, a call to detail will only introduce a if-branch which is much
  cheaper than a function call.
2019-04-05 13:12:19 -07:00
Jingyu Zhou 38c6681349 Fix some signed and unsigned mismatch warnings. 2019-03-26 14:54:11 -07:00
Vishesh Yadav c37291a366 fix: Use '_' instead of ':' in IPv6 tracefile names
':' is not acceptable on Windows. Reason to choose '_' instead of '.'
is to differentiate b/w IPv4 and IPv6 easily, and also '..1' in
filename looks weirder than '__1', which would happen with shortened
IPv6 addresses. And non-shortened IPv6 addreses are just lots of 0s.
2019-03-20 14:00:33 -07:00
Vishesh Yadav 592e224155 net: add/use formatIpPort to format IP:PORT pairs #963 2019-03-04 14:12:45 -08:00
Vishesh Yadav 25daabdc02 net: TraceEvent and toIPVectorString for new IPAddress structure #963 2019-03-04 14:12:45 -08:00
Vishesh Yadav 57832e625d net: Support IPv6 #963
- NetworkAddress now contains IPAddress object which can be either
IPv4 or IPv6 address. 128bits are used even for IPv4 addresses,
however only 32bits are used when using/serializing IPv4 address.

- ConnectPacket is updated to store IPv6 address. Backward compatible
with old format since the first 32bits of IP address field is used
for serialization of IPv4.

- Mainly updates rest of the code to use IPAddress structure instead
of plain uint32_t.

- IPv6 address/pair ports should be represented as `[ip]:port` as per
convention. This applies to both cluster files and command line
arguments.
2019-03-04 14:12:41 -08:00
A.J. Beamon 655c9d82c7 Various cleanup from review 2019-03-01 14:06:47 -08:00
A.J. Beamon eb629d87a5 Add information about batch ratekeeper to status. Make it possible to track latencies in the ReadWrite workload for concurrently run instances separately. 2019-02-28 09:53:16 -08:00
Andrew Noyes e055fdab03 Better diagnostics for unrecognized trace format 2019-01-29 13:08:47 -08:00
Andrew Noyes 3f399dd9be Indent within anonymous namespace 2019-01-03 09:21:09 -08:00
Andrew Noyes 80dd1efaf6 Assign to g_traceLog.formatter 2019-01-02 16:42:00 -08:00
anoyes b8df5acc15 Add --trace_format flag to fdbserver 2018-12-20 15:02:01 -08:00
anoyes 33a4aac249 Fix build 2018-12-20 11:55:59 -08:00
Markus Pilman cdc7e83993
Update flow/Trace.cpp
Co-Authored-By: alexmiller-apple <35046903+alexmiller-apple@users.noreply.github.com>
2018-12-14 11:30:40 -08:00
Markus Pilman 40890e9dbe Implemented a json log formatter 2018-12-13 21:46:02 -08:00
Robert Escriva 268093a96d Adjust all includes to be relative to the root.
Remove the use of relative paths.  A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h".  Adjust so that every include references such a header with the
latter form.

Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
A.J. Beamon 9e79f9ec59 Cleanup from review 2018-09-05 15:53:12 -07:00
A.J. Beamon 2de0b5d6d7 Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations. 2018-09-05 15:06:14 -07:00
A.J. Beamon 168dce94cb Remove some trace event suppressions that were happening off the network thread. Downgrade some trace events related to trace logging problems from SevError to SevWarnAlways. 2018-08-14 09:00:43 -07:00
Evan Tschannen 1c29275672 call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details. 2018-08-01 14:30:57 -07:00
Evan Tschannen 5054c194e2 Some trace events are logged before FLOW_KNOBS are initialized 2018-07-01 14:30:37 -04:00
A.J. Beamon a680837ee4 The Machine field was missing in early trace events. The logGroup field was not being properly set. 2018-06-28 15:28:58 -07:00
A.J. Beamon 1ff42e078f Add missing include for Windows, remove throw from TraceEvent destructor. 2018-06-28 14:59:23 -07:00
A.J. Beamon cbc840ad0a Move the spammy trace event backstop from the destructor to the constructor. This allows us to avoid doing needless work on a trace event that is going to be throttled. 2018-06-27 15:51:30 -07:00
A.J. Beamon fe956bc35a Address review comments 2018-06-26 14:37:21 -07:00
A.J. Beamon f965954122 Merge commit '82be52205b95464e355c449fdf3e7d483fa06677' into trace-log-refactor
# Conflicts:
#	fdbserver/Status.actor.cpp
#	fdbserver/workloads/DDMetrics.actor.cpp
#	flow/Trace.cpp
2018-06-08 16:22:22 -07:00
A.J. Beamon 06ccd9a500 Allow trace event type names to end with an underscore. 2018-06-08 15:49:31 -07:00
A.J. Beamon 1fdfe20908 Relax the rules on trace event Types a bit by allowing multiple underscores, as well as starting with an underscore and consecutive underscores. 2018-06-08 15:40:29 -07:00
A.J. Beamon 0ca51989bb Merge branch 'master' into trace-log-refactor
# Conflicts:
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/Status.actor.cpp
#	flow/Trace.cpp
2018-06-08 13:24:30 -07:00
A.J. Beamon e5488419cc Attempt to normalize trace events:
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.

Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.

This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
A.J. Beamon 3ea5fc72f0 Rename trace file logGroup attribute to LogGroup 2018-05-31 15:34:17 -07:00
A.J. Beamon f1dea3bed9 Rework the formatter a bit and extract out the writing logic so that it could be pluggable. 2018-05-31 13:27:35 -07:00
A.J. Beamon f7735d000f Fix linking issue in flow bindings by using a different variant of printable(). 2018-05-31 11:16:01 -07:00
A.J. Beamon 78839b20fd Merge branch 'master' into trace-log-refactor
# Conflicts:
#	flow/Trace.cpp
2018-05-31 10:46:20 -07:00
A.J. Beamon e2e69b5632 Rework how rolled events are logged. Not very efficient, but it doesn't modify the cached version returned in queries. 2018-05-11 11:24:20 -07:00
A.J. Beamon fe1c9d5be4 Make the trace log formatter reference counted. 2018-05-09 14:35:15 -07:00
A.J. Beamon 047563aa62 Support trace event overflow. 2018-05-04 15:19:56 -07:00
A.J. Beamon 2a46fdb31d Change formatv to vsformat, which now returns the length of the string (or <0 for error) and takes the output string as an argument. 2018-05-04 13:35:25 -07:00
A.J. Beamon ce0c991e78 Refactor trace events to store a vector of fields that aren't encoded until write time. Better support for pre-network trace events. Rework how trace events are queried. Some initial work towards pluggable formatting of logs. 2018-05-02 10:44:38 -07:00
A.J. Beamon 62726efa82 Add support for logging trace events before the network is created and the trace log opened. 2018-03-28 08:21:22 -07:00
Evan Tschannen 5db52ab081
Merge pull request #87 from etschannen/feature-remote-logs
Feature remote logs
2018-03-23 12:55:17 -07:00
Evan Tschannen 59723f51f8 fix: continue to attempt to lock logs until remote logs are recovered, this is so that remote logs get locked and readers know they will not have any more data
do not throttle trace events in simulation
2018-03-14 12:39:55 -07:00
Alec Grieser 70a05c1a9b
fix some compiler whinges 2018-03-13 15:00:16 -07:00