131 lines
5.3 KiB
HTML
131 lines
5.3 KiB
HTML
<html devsite>
|
|
<head>
|
|
<title>Interpreting CTS results</title>
|
|
<meta name="project_path" value="/_project.yaml" />
|
|
<meta name="book_path" value="/_book.yaml" />
|
|
</head>
|
|
<body>
|
|
<!--
|
|
Copyright 2017 The Android Open Source Project
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|
you may not use this file except in compliance with the License.
|
|
You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
-->
|
|
|
|
|
|
|
|
<p>The CTS test results are placed in the file:</p>
|
|
<pre class="devsite-click-to-copy">
|
|
$CTS_ROOT/android-cts/repository/results/<start_time>.zip
|
|
</pre>
|
|
|
|
<p>If you have built the CTS yourself, <em>$CTS_ROOT</em> will resemble the
|
|
path <em>out/host/linux-x86/cts</em> but differ by platform. This reflects the
|
|
path where you have uncompressed the prebuilt official CTS <a
|
|
href="downloads.html">downloaded</a> from this site.</p>
|
|
|
|
<p>Inside the zip, the testResult.xml file contains the actual results. Open
|
|
this file in any web browser (HTML5 compatible browser recommended) to view the
|
|
test results.</p>
|
|
|
|
<p>If testResult.xml displays a blank page when using the Chrome browser,
|
|
<a href="https://www.chromium.org/developers/how-tos/run-chromium-with-flags">change
|
|
your browser configuration</a> to enable the
|
|
<em>--allow-file-access-from-files</em> command line flag.</p>
|
|
|
|
<h3 id="reading-the-test-results">Reading the test results</h3>
|
|
|
|
<p>The details of the test results depend on which version of CTS you are
|
|
using:</p>
|
|
|
|
<ul>
|
|
<li>CTS v1 for Android 6.0 and earlier</a>
|
|
<li>CTS v2 for Android 7.0 and later</a>
|
|
</ul>
|
|
|
|
<p class="note"><strong>Note:</strong> The results are provided to help you
|
|
ensure the software remains compatible throughout the development process and
|
|
act as a common format for communicating the compatibility status of your
|
|
device with other parties.</p>
|
|
|
|
<h4 id="device-information">Device Information</h4>
|
|
|
|
<p>In CTS v6.0 and earlier, select Device Information (link above Test Summary) to
|
|
view details about the device, firmware (make, model, firmware build,
|
|
platform), and device hardware (screen resolution, keypad, screen type). (CTS
|
|
v7.0 does not display device information.)</p>
|
|
|
|
<h4 id="test-summary">Test Summary</h4>
|
|
|
|
<p>The <em>Test Summary</em> section provides executed test plan details, like
|
|
the CTS plan name and execution start and end times. It also presents an
|
|
aggregate summary of the number of tests that passed, failed, timed out, or
|
|
could not be executed.</p>
|
|
|
|
<section class="expandable">
|
|
<h4 class="showalways">Show CTS v1 sample test summary</h4>
|
|
<img src="/compatibility/cts/images/cts-test-summary.png" alt="CTS v1 test summary" id="figure1a" />
|
|
<p class="img-caption"><strong>Figure 1a.</strong> CTS v1 sample test summary </p>
|
|
</section>
|
|
|
|
|
|
<section class="expandable">
|
|
<h4 class="showalways">Show CTS v2 sample test summary</h4>
|
|
<img src="/compatibility/cts/images/cts-v2-test-summary.png" alt="CTS v2 test summary" id="figure1b" />
|
|
<p class="img-caption"><strong>Figure 1b.</strong> CTS v2 sample test summary </p>
|
|
</section>
|
|
|
|
<p> </p>
|
|
|
|
|
|
<h4 id="test-report">Test Report</h4>
|
|
<p>The next section, the CTS test report, provides a summary of tests passed per package.</p>
|
|
|
|
<p>This is followed by details of the actual tests that were executed. The
|
|
report lists the test package, test suite, test case, and the executed tests.
|
|
It shows the result of the test execution—pass, fail, timed out, or not
|
|
executed. In the event of a test failure details are provided to help diagnose
|
|
the cause.</p>
|
|
|
|
<p>Further, the stack trace of the failure is available in the XML file but is
|
|
not included in the report to ensure brevity—viewing the XML file with a text
|
|
editor should provide details of the test failure (search for the
|
|
<em><Test></em> tag corresponding to the failed test and look within it
|
|
for the <em><StackTrace></em> tag).</p>
|
|
|
|
<section class="expandable">
|
|
<h4 class="showalways">Show CTS v1 sample test report</h4>
|
|
<img src="/compatibility/cts/images/cts-test-report.png" alt="CTS v1 test report" id="figure2a" />
|
|
<p class="img-caption"><strong>Figure 2a.</strong> CTS v1 sample test report</p>
|
|
</section>
|
|
|
|
<section class="expandable">
|
|
<h4 class="showalways">Show CTS v2 sample test report</h4>
|
|
<img src="/compatibility/cts/images/cts-v2-test-report.png" alt="CTS v2 test report" id="figure2b" />
|
|
<p class="img-caption"><strong>Figure 2b.</strong> CTS v2 sample test report</p>
|
|
</section>
|
|
|
|
<p> </p>
|
|
|
|
<h3 id="incomplete-test-modules">Reviewing test_result.xml for incomplete test modules</h3>
|
|
|
|
<p>
|
|
To determine the number of incomplete modules in a given test session, run
|
|
command 'list results'. The count of Modules Completed and Total Modules are
|
|
listed for each previous session. To determine which modules are complete vs.
|
|
incomplete, open the test_result.xml file and read the value of the "done"
|
|
attribute for each module in the result report. Modules with value done =
|
|
"false" have not run to completion.
|
|
</p>
|
|
|
|
</body>
|
|
</html>
|