166 lines
7.8 KiB
Markdown
166 lines
7.8 KiB
Markdown
<!-- Copyright 2016 The Chromium Authors. All rights reserved.
|
|
Use of this source code is governed by a BSD-style license that can be
|
|
found in the LICENSE file.
|
|
-->
|
|
|
|
Timeline-Based Measurement v2 is a system for computing metrics from traces.
|
|
|
|
A TBM2 metric is a Javascript function that takes a trace Model and produces
|
|
Histograms.
|
|
|
|
|
|
Coding Practices
|
|
==============
|
|
|
|
Please follow the [Catapult Javascript style guide](/docs/style-guide.md) so
|
|
that the TBM2 maintainers can refactor your metric when we need to update the TBM2
|
|
API.
|
|
|
|
Please write a unit test for your metric.
|
|
|
|
If your metric computes information from the trace that may be of general use to
|
|
other metrics or the trace viewer UI, then the TBM2 maintainers may ask for your
|
|
help to generalize your innovation into a part of the Trace Model such as the
|
|
[UserModel](/tracing/tracing/model/user_model/user_model.html) or
|
|
[ModelHelpers](/tracing/tracing/model/helpers/chrome_browser_helper.html).
|
|
|
|
Use the dev server to develop and debug your metric.
|
|
|
|
* Run `./bin/run_dev_server`
|
|
* Navigate to
|
|
[http://localhost:8003/tracing_examples/trace_viewer.html](http://localhost:8003/tracing_examples/trace_viewer.html).
|
|
* Open a trace that your metric can be computed from.
|
|
* Open the Metrics side panel on the right.
|
|
* Select your metric from the drop-down.
|
|
* Inspect the results and change your metric if necessary.
|
|
* Open different traces to explore corner cases in your metric.
|
|
|
|
|
|
Trace Model
|
|
===========
|
|
|
|
Trace logs are JSON files produced by tracing systems in Chrome, Android, linux
|
|
perf, BattOr, etc. The trace model is an object-level representation of events
|
|
parsed from a trace log. The trace model contains Javascript objects
|
|
representing
|
|
|
|
* OS [processes](/tracing/tracing/model/process.html),
|
|
[threads](/tracing/tracing/model/thread.html),
|
|
* utilities for finding special processes and threads such as
|
|
[ChromeBrowserHelper](/tracing/tracing/model/helpers/chrome_browser_helper.html),
|
|
* synchronous [ThreadSlices](/tracing/tracing/model/thread_slice.html)
|
|
and asynchronous [AsyncSlices](/tracing/tracing/model/async_slice.html),
|
|
* [snapshots](/tracing/tracing/model/object_snapshot.html) of object state as it changes throughout time,
|
|
* [RPCs](/tracing/tracing/model/flow_event.html),
|
|
* [FrameBlameContexts](/tracing/tracing/extras/chrome/blame_context/blame_context.html),
|
|
* battery [power samples](/tracing/tracing/model/power_sample.html),
|
|
* synthetic higher-level abstractions representing complex sets of
|
|
events such as
|
|
[UserExpectations](/tracing/tracing/model/user_model/user_expectation.html),
|
|
* and [more](/tracing/tracing/model/model.html)!
|
|
|
|
|
|
Histograms
|
|
==========
|
|
|
|
A [Histogram](/tracing/tracing/value/histogram.html) is basically a common
|
|
[histogram](https://en.wikipedia.org/wiki/Histogram), but with a few extra bells
|
|
and whistles that are particularly useful for TBM2 metrics.
|
|
|
|
* Specify units of samples and improvement direction with
|
|
[Unit](/tracing/tracing/value/unit.html)
|
|
* JSON serialization with asDict()/fromDict()
|
|
* Build custom bin boundaries with HistogramBinBoundaries
|
|
* Compute statistics such as average, stddev, sum, and percentiles
|
|
* Customize which statistics are serialized with customizeSummaryOptions()
|
|
* Count non-numeric samples
|
|
* Store a random subset of sample values
|
|
* getDifferenceSignificance() computes whether two histograms are significantly
|
|
different with a Mann-Whitney U hypothesis test
|
|
* addHistogram() merges two Histograms with the same units and bin boundaries
|
|
|
|
But the most complex special feature of Histograms is their Diagnostics.
|
|
|
|
|
|
Diagnostics
|
|
===========
|
|
|
|
When a metric significantly regresses, you then need to diagnose why it
|
|
regressed. Diagnostics are pieces of information that metrics attach to
|
|
Histograms in order help you diagnose regressions. Diagnostics may be associated
|
|
either with the entire Histogram directly, or with a particular sample.
|
|
|
|
Attach a Diagnostic to a Histogram:
|
|
|
|
```javascript
|
|
histogram.diagnostics.set('name', diagnostic)
|
|
// or
|
|
values.addHistogram(histogram, {name: diagnostic})
|
|
```
|
|
|
|
Attach a Diagnostic to a sample:
|
|
|
|
```javascript
|
|
histogram.addSample(number, {name: diagnostic})
|
|
```
|
|
|
|
The types of Diagnostics are
|
|
|
|
* [Generic](/tracing/tracing/value/diagnostics/generic.html): This can contain
|
|
any data that can be serialized and deserialized using JSON.stringify() and
|
|
JSON.parse(), including numbers, strings, Arrays, and dictionaries (simple
|
|
Objects). It will be visualized using
|
|
[generic-object-view](/tracing/tracing/ui/analysis/generic_object_view.html),
|
|
which is quite smart about displaying tabular data using tables, URLs using
|
|
HTML anchor tags, pretty-printing, recursive data structures, and more.
|
|
* [RelatedEventSet](/tracing/tracing/value/diagnostics/related_event_set.html):
|
|
This is a Set of references to Events in the trace model. Visually, they
|
|
are displayed as HTML links which, when clicked in the metrics-side-panel,
|
|
select the referenced Events in the trace viewer's timeline view. When
|
|
clicked in results2.html, they currently do nothing, but should eventually
|
|
open the trace that contains the events and select them.
|
|
* [Breakdown](/tracing/tracing/value/diagnostics/breakdown.html):
|
|
Structurally, these are Maps from strings to numbers. Conceptually, they
|
|
describe what fraction of a whole (either a Histogram or a sample) is due to
|
|
some sort of category - either a category of event, CPU sample, memory
|
|
consumer, whathaveyou. Visually, they are a stacked bar chart with a single
|
|
bar, which is spiritually a pie chart, but less misleading.
|
|
* [RelatedValueSet](/tracing/tracing/value/diagnostics/related_value_set.html):
|
|
These are Sets of references to other Histograms. Visually, they are a set
|
|
of HTML links which, when clicked, select the contained Histograms. The text
|
|
content of the HTML link is the name of the referenced Histogram.
|
|
* [RelatedValueMap](/tracing/tracing/value/diagnostics/related_value_map.html):
|
|
These are Maps from strings to references to other Histograms. Visually, they
|
|
are a set of HTML links similar to RelatedValueSet, but the text content of
|
|
the link is the Map's string key instead of the Histogram's name. One example
|
|
application is when a Histogram was produced not directly by a metric, but
|
|
rather by merging together other Histograms, then it will have a
|
|
RelatedValueMap named 'merged from' that refers to the Histograms that were
|
|
merged by their grouping key, e.g. the telemetry story name.
|
|
* [RelatedHistogramBreakdown](/tracing/tracing/value/diagnostics/related_histogram_breakdown.html):
|
|
Structurally, this is a RelatedValueMap, but conceptually and visually, this
|
|
is a Breakdown. Whereas Breakdown's stacked bar chart derives its data from
|
|
the numbers contained explicitly in the Breakdown, a
|
|
RelatedHistogramBreakdown's stacked
|
|
bar chart derives its data from the referenced Histograms' sums.
|
|
* [IterationInfo](/tracing/tracing/value/diagnostics/iteration_info.html):
|
|
This is automatically attached to every Histogram produced by telemetry.
|
|
Structurally, it's a class with explicit named fields.
|
|
Conceptually, it contains information about the origins of the trace that was
|
|
consumed by the metric that produced the Histogram, such as the benchmark
|
|
name, story name, benchmark start timestamp, OS version, Chrome version, etc.
|
|
Visually, IterationInfos are displayed as a table.
|
|
|
|
|
|
Consumers of Histograms
|
|
=======================
|
|
|
|
Histograms are consumed by
|
|
|
|
* [value-set-table](/tracing/tracing/value/ui/value-set-table.html) in both
|
|
results2.html and the Metrics side panel in trace viewer,
|
|
* the [dashboard](https://chromeperf.appspot.com) indirectly via their statistics.
|
|
|
|
Currently, telemetry discards Histograms and Diagnostics, and only passes their
|
|
statistics scalars to the dashboard. Histograms and their Diagnostics will be
|
|
passed directly to the dashboard early 2017.
|