fuquan/allwinner_a64
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

upload android base code part6

Browse source
This commit is contained in:
August 2018-08-08 17:48:24 +08:00
parent 421e214c7d
commit 4e516ec6ed
35396 changed files with 9188716 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

40
android/platform_testing/docs/development/index.md Normal file
View file

@ -0,0 +1,40 @@
# Test Development Workflow
To integrate tests into platform continuous testing service, they should meet
the following guidelines.
## Test Types
Supported test types are:
* standard [instrumentation](http://developer.android.com/tools/testing/testing_android.html) tests
* supports both functional and metrics tests
* native tests
* functional: [gtest](https://github.com/google/googletest) framework
* metrics: native benchmark tests using [google-benchmark](https://github.com/google/benchmark)
Functional tests make assertions of pass or fail on test cases, while metrics
tests generally performs an action repeatedly to collect timing metrics.
With standardized input/output format, the need for customized result parsing
and post-processing per test is eliminated, and generic test harnesses can be
used for all tests that fit into the convention.
### Test Case Guidelines
Test cases executed via continuous testing service are expected to be
**hermetic**:
* no Google account sign-in
* no connectivity setup (telephony/wifi/bluetooth/NFC)
* no test parameters passed in
* no setup or tear down performed by test harness for a specific test case
### Building Tests
If you are new to the workflow of adding and executing tests, please see:
* [Instrumentation Tests](instrumentation.md) (supports both functional and
metrics tests)
* [Native Tests](native.md)
* [Native Metric Tests](metrics.md)

383
android/platform_testing/docs/development/instr-app-e2e.md Normal file
View file

@ -0,0 +1,383 @@
# Instrumentation Targeting an Application: A Complete Example
[TOC]
If you are new to Android platform development, you might find this complete
example of adding a brand new instrumentation test from scratch useful to
demonstrate the typical workflow involved.
Note that this guide assumes that you already have some knowledge in the
platform source tree workflow. If not, please refer to
https://source.android.com/source/requirements. The example
covered here is writing an new instrumentation test with target package set at
its own test application package. If you are unfamiliar with the concept, please
read through the [testing basics](../basics/index.md) page.
This guide uses the follow test to serve as an sample:
* frameworks/base/packages/Shell/tests
It's recommended to browse through the code first to get a rough impression
before proceeding.
## Deciding on a Source Location
Because the instrumentation test will be targeting an application, the convention
is to place the test source code in a `tests` directory under the root of your
component source directory in platform source tree.
See more discussions about source location in the [end-to-end example for
self-instrumenting tests](instr-self-e2e.md#location).
## Makefile
Each new test module must have a makefile to direct the build system with module
metadata, compile time depdencies and packaging instructions.
frameworks/base/packages/Shell/tests/Android.mk
A snapshot is included here for convenience:
```makefile
LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE_TAGS := tests
LOCAL_SRC_FILES := $(call all-java-files-under, src)
LOCAL_JAVA_LIBRARIES := android.test.runner
LOCAL_STATIC_JAVA_LIBRARIES := ub-uiautomator junit legacy-android-test
LOCAL_PACKAGE_NAME := ShellTests
LOCAL_INSTRUMENTATION_FOR := Shell
LOCAL_COMPATIBILITY_SUITE := device-tests
LOCAL_CERTIFICATE := platform
include $(BUILD_PACKAGE)
```
Some select remarks on the makefile:
```makefile
LOCAL_MODULE_TAGS := tests
```
This setting declares the module as a test module, which will instruct the build
system to automatically skip proguard stripping, since that's typically
problematic for tests.
```makefile
LOCAL_CERTIFICATE := platform
```
This setting instructs the build system to sign the test application package
with the platform certificate. This is because for a test application package to
be able to instrument on the targeted application package, these two packages
must be signed with the same certificate; otherwise allowing packages to be
instrumented on arbitrarily would be a security concern. To find out the signing
certificate of the application packge you are testing, look for
`LOCAL_CERTIFICATE` in its `Android.mk`; and if there isn't one, simply skip
this field in your test application makefile as well.
```makefile
LOCAL_JAVA_LIBRARIES := android.test.runner
```
This setting tells the build system to put Java library `android.test.runner` on
classpath during compilation, as opposed to statically incorporating the library
into the current package. This is typically done for Java code that is
referenced by the code in current package, and will be automatically placed on
package classpath at runtime. In the context of tests for application, strictly
speaking, both framework APIs and code in application under test fall into this
category, however, the former is done via implicit rules by build system at
compile time and by framework at runtime, and the latter is done via
`LOCAL_INSTRUMENTATION_FOR` (see below) at compile time and via
`android:targetPackage` (see below) in manifest by instrumentation framework at
runtime.
```makefile
LOCAL_STATIC_JAVA_LIBRARIES := ub-uiautomator junit legacy-android-test
```
This setting instructs the build system to incorporate the contents of the named
modules into the resulting apk of current module. This means that each named
module is expected to produce a `.jar` file, and its content will be used for
resolving classpath references during compile time, as well as incorporated into
the resulting apk.
The platform source tree also included other useful testing frameworks such as
`ub-uiautomator`, `easymock` and so on.
```makefile
LOCAL_PACKAGE_NAME := ShellTests
```
This setting is required if `BUILD_PACKAGE` when used later: it gives a name to
your module, and the resulting apk will be named the same and with a `.apk`
suffix, e.g. in this case, resulting test apk is named as
`ShellTests.apk`. In addition, this also defines a make target name
for your module, so that you can use `make [options] <LOCAL_PACKAGE_NAME>` to
build your test module and all its dependencies.
```makefile {# LOCAL_INSTRUMENTATION_FOR}
LOCAL_INSTRUMENTATION_FOR := Shell
```
As mentioned, during execution of an instrumentation test, the application under
test is restarted with the instrumentation code injected for execution. The test
can reference any classes and its instances of the application under test. This
means that the test code may contain references to classes defined by the
application under test, so during compile time, the build system needs to
properly resolve such references. This setting provides the module name of
application under test, which should match the `LOCAL_PACKAGE_NAME` of in the
makefile for your application. At compile time, the build system will try to
look up the intermediate files for the named module, and use them on the
classpath for the Java compiler.
```makefile
LOCAL_COMPATIBILITY_SUITE := device-tests
```
This line builds the testcase as part of the device-tests suite, which is
meant to target a specific device and not a general ABI. If only the ABI
needs to be targetted, it can be swapped with 'general-tests'.
```makefile
include $(BUILD_PACKAGE)
```
This includes a core makefile in build system that performs the necessary steps
to generate an apk based on the settings provided by the preceding variables.
The generated apk will be named after `LOCAL_PACKAGE_NAME`, e.g.
`SettingsGoogleUnitTests.apk`. And if `tests` is used as `LOCAL_MODULE_TAGS` and
there are no other customizations, you should be able to find your test apk in:
* `${OUT}/data/app/<LOCAL_PACKAGE_NAME>/<LOCAL_PACKAGE_NAME>.apk`
e.g. `${OUT}/data/app/ShellTests/ShellTests.apk`
## Manifest file
Just like a regular application, each instrumentation test module needs a
manifest file. If you name the file as `AndroidManifest.xml` and provide it next
to `Android.mk` for your test tmodule, it will get included automatically by the
`BUILD_PACKAGE` core makefile.
Before proceeding further, it's highly recommended to go through the external
[documentation on manifest file](https://developer.android.com/guide/topics/manifest/manifest-intro.html)
first.
This gives an overview of basic components of a manifest file and their
functionalities.
Latest version of the manifest file for the sample gerrit change can be accessed
at:
https://android.googlesource.com/platform/frameworks/base/+/master/packages/Shell/tests/AndroidManifest.xml
A snapshot is included here for convenience:
```xml
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.android.shell.tests">
<application>
<uses-library android:name="android.test.runner" />
<activity
android:name="com.android.shell.ActionSendMultipleConsumerActivity"
android:label="ActionSendMultipleConsumer"
android:theme="@android:style/Theme.NoDisplay"
android:noHistory="true"
android:excludeFromRecents="true">
<intent-filter>
<action android:name="android.intent.action.SEND_MULTIPLE" />
<category android:name="android.intent.category.DEFAULT" />
<data android:mimeType="*/*" />
</intent-filter>
</activity>
</application>
<instrumentation android:name="android.support.test.runner.AndroidJUnitRunner"
android:targetPackage="com.android.shell"
android:label="Tests for Shell" />
</manifest>
```
Some select remarks on the manifest file:
```xml
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.android.shell.tests">
```
The `package` attribute is the application package name: this is the unique
identifier that the Android application framework uses to identify an
application (or in this context: your test application). Each user in the system
can only install one application with that package name.
Since this is a test application package, independent from the application
package under test, a different package name must be used: one common convention
is to add a suffix `.test`.
Furthermore, this `package` attribute is the same as what
[`ComponentName#getPackageName()`](https://developer.android.com/reference/android/content/ComponentName.html#getPackageName\(\))
returns, and also the same you would use to interact with various `pm` sub
commands via `adb shell`.
Please also note that although the package name is typically in the same style
as a Java package name, it actually has very few things to do with it. In other
words, your application (or test) package may contain classes with any package
names, though on the other hand, you could opt for simplicity and have your top
level Java package name in your application or test identical to the application
package name.
```xml
<uses-library android:name="android.test.runner" />
```
This is required for all Instrumentation tests since the related classes are
packaged in a separate framework jar library file, therefore requires additional
classpath entries when the test package is invoked by application framework.
```xml
android:targetPackage="com.android.shell"
```
This sets the target package of the instrumentation to `com.android.shell.tests`.
When the instrumentation is invoked via `am instrument` command, the framework
restarts `com.android.shell.tests` process, and injects instrumentation code into
the process for test execution. This also means that the test code will have
access to all the class instances running in the application under test and may
be able to manipulate state depends on the test hooks exposed.
## Test Configuration File
In order to simplify test execution, you also need write a test configuration
file for Android's test harness, [TradeFederation](https://source.android.com/devices/tech/test_infra/tradefed/).
The test configuration can specify special device setup options and default
arguments to supply the test class.
The config can be found:
frameworks/base/packages/Shell/tests/src/com/android/shell/BugreportReceiverTest.javast.java
A snapshot is included here for convenience:
```xml
<configuration description="Runs Tests for Shell.">
<target_preparer class="com.android.tradefed.targetprep.TestAppInstallSetup">
<option name="test-file-name" value="ShellTests.apk" />
</target_preparer>
<option name="test-suite-tag" value="apct" />
<option name="test-tag" value="ShellTests" />
<test class="com.android.tradefed.testtype.AndroidJUnitTest" >
<option name="package" value="com.android.shell.tests" />
<option name="runner" value="android.support.test.runner.AndroidJUnitRunner" />
</test>
</configuration>
```
Some select remarks on the test configuration file:
```xml
<target_preparer class="com.android.tradefed.targetprep.TestAppInstallSetup">
<option name="test-file-name" value="ShellTests.apk"/>
</target_preparer>
```
This tells TradeFederation to install the ShellTests.apk onto the target
device using a specified target_preparer. There are many target preparers
available to developers in TradeFederation and these can be used to ensure
the device is setup properly prior to test execution.
```xml
<test class="com.android.tradefed.testtype.AndroidJUnitTest">
<option name="package" value="com.android.shell.tests"/>
<option name="runner" value="android.support.test.runner.AndroidJUnitRunner"/>
</test>
```
This specifies the TradeFederation test class to use to execute the test and
passes in the package on the device to be executed and the test runner
framework which is JUnit in this case.
Look here for more information on [Test Module Configs](../test-config.md)
## JUnit4 Features
Using `android-support-test` library as test runner enables adoptation of new
JUnit4 style test classes, and the sample gerrit change contains some very basic
use of its features.
Latest source code for the sample gerrit change can be accessed at:
frameworks/base/packages/Shell/tests/src/com/android/shell/BugreportReceiverTest.javast.java
While testing patterns are usually specific to component teams, there are some
generally useful usage patterns.
```java
@SmallTest
@RunWith(AndroidJUnit4.class)
public final class FeatureFactoryImplTest {
```
A significant difference in JUnit4 is that tests are no longer required to
inherit from a common base test class; instead, you write tests in plain Java
classes and use annotation to indicate certain test setup and constraints. In
this example, we are instructing that this class should be run as an Android
JUnit4 test.
The `@SmallTest` annotation specified a test size for the entire test class: all
test methods added into this test class inherit this test size annotation.
pre test class setup, post test tear down, and post test class tear down:
similar to `setUp` and `tearDown` methods in JUnit4.
`Test` annotation is used for annotating the actual test.
**Important**: the test methods themselves are annotated with `@Test`
annotation; and note that for tests to be executed via APCT, they must be
annotated with test sizes. Such annotation may be applied at method scope, or
class scope.
```java
@Before
public void setup() {
...
@Test
public void testGetProvider_shouldCacheProvider() {
...
```
The `@Before` annotation is used on methods by JUnit4 to perform pre test setup.
Although not used in this example, there's also `@After` for post test teardown.
Similarly, the `@BeforeClass` and `@AfterClass` annotations are can be used on
methods by JUnit4 to perform setup before executing all tests in a test class,
and teardown afterwards. Note that the class-scope setup and teardown methods
must be static.
As for the test methods, unlike in earlier version of JUnit, they no longer need
to start the method name with `test`, instead, each of them must be annotated
with `@Test`. As usual, test methods must be public, declare no return value,
take no parameters, and may throw exceptions.
```java
Context context = InstrumentationRegistry.getTargetContext();
```
Because the JUnit4 tests no longer require a common base class, it's no longer
necessary to obtain `Context` instances via `getContext()` or
`getTargetContext()` via base class methods; instead, the new test runner
manages them via [`InstrumentationRegistry`](https://developer.android.com/reference/android/support/test/InstrumentationRegistry.html)
where contextual and environmental setup created by instrumentation framework is
stored. Through this class, you can also call:
* `getInstrumentation()`: the instance to the `Instrumentation` class
* `getArguments()`: the command line arguments passed to `am instrument` via
`-e <key> <value>`
## Build & Test Locally
Follow these [Instructions](../instrumentation.md)

394
android/platform_testing/docs/development/instr-self-e2e.md Normal file
View file

@ -0,0 +1,394 @@
# Self-Instrumenting Tests: A Complete Example
[TOC]
If you are new to Android platform development, you might find this complete
example of adding a brand new instrumentation test from scratch useful to
demonstrate the typical workflow involved.
Note that this guide assumes that you already have some knowledge in the
platform source tree workflow. If not, please refer to
https://source.android.com/source/requirements. The example
covered here is writing an new instrumentation test with target package set at
its own test application package. If you are unfamiliar with the concept, please
read through the [testing basics](../basics/index.md) page.
This guide uses the follow test to serve as an sample:
* [Hello World Instrumentation Test](../../tests/example/instrumentation)
It's recommended to browse through the code first to get a rough impression
before proceeding.
## Deciding on a Source Location {#location}
Typically your team will already have an established pattern of places to check
in code, and places to add tests. Most team owns a single git repository, or
share one with other teams but have a dedicated sub directory that contains
component source code.
Assuming the root location for your component source is at `<component source
root>`, most components have `src` and `tests` folders under it, and some
additional files such as `Android.mk` (or broken up into additional `.mk` files),
the manifest file `AndroidManifest.xml`, and the test configuration file
'AndroidTest.xml'.
Since you are adding a brand new test, you'll probably need to create the
`tests` directory next to your component `src`, and populate it with content.
In some cases, your team might have further directory structures under `tests`
due to the need to package different suites of tests into individual apks. And
in this case, you'll need to create a new sub directory under `tests`.
Regardless of the structure, you'll end up populating the `tests` directory or
the newly created sub directory with files similar to what's in
`instrumentation` directory in the sample gerrit change. The sections below will
explain in further details of each file.
## Makefile
Each new test module must have a makefile to direct the build system with module
metadata, compile time depdencies and packaging instructions.
[Latest version of the makefile](../../tests/example/instrumentation/Android.mk)
A snapshot is included here for convenience:
```makefile
LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_SRC_FILES := $(call all-java-files-under, src)
LOCAL_MODULE_TAGS := tests
LOCAL_PACKAGE_NAME := HelloWorldTests
LOCAL_STATIC_JAVA_LIBRARIES := android-support-test
LOCAL_CERTIFICATE := platform
LOCAL_COMPATIBILITY_SUITE := device-tests
include $(BUILD_PACKAGE)
```
Some select remarks on the makefile:
```makefile
LOCAL_MODULE_TAGS := tests
```
This setting declares the module as a test module, which will instruct the build
system to automatically skip proguard stripping, since that's typically
problematic for tests.
```makefile
LOCAL_PACKAGE_NAME := HelloWorldTests
```
This setting is required when `BUILD_PACKAGE` is used later: it gives a name to
your module, and the resulting apk will be named the same and with a `.apk`
suffix, e.g. in this case, resulting test apk is named as `HelloWorldTests.apk`.
In addition, this also defines a make target name for your module, so that you
can use `make [options] <LOCAL_PACKAGE_NAME>` to build your test module and all
its dependencies.
```makefile
LOCAL_STATIC_JAVA_LIBRARIES := android-support-test
```
This setting instructs the build system to incorporate the contents of the named
modules into the resulting apk of current module. This means that each named
module is expected to produce a `.jar` file, and its content will be used for
resolving classpath references during compile time, as well as incorporated into
the resulting apk.
In this example, things that might be generally useful for tests:
* `android-support-test` is the prebuilt for Android Test Support Library,
which included the new test runner `AndroidJUnitRunner`: a replacement for
the now deprecated built-in `InstrumentationTestRunner`, with support for
JUnit4 testing framework. Find out more at:
* https://google.github.io/android-testing-support-library/
If you are building a new instrumentation module, you should always start
with this library as your test runner.
The platform source tree also included other useful testing frameworks such as
`ub-uiautomator`, `mockito-target`, `easymock` and so on.
```makefile
LOCAL_CERTIFICATE := platform
```
This setting instructs the build system to sign the apk with the same
certificate as the core platform. This is needed if your test uses a signature
protected permission or API. Note that this is suitable for platform continuous
testing, but should *not* be used in CTS test modules. Note that this example
uses this certificat setting only for the purpose of illustration: the test code
of the example does not actually need for the test apk to be signed with the
special platform certificate.
If you are writing an instrumentation for your component that lives outside of
system server, that is, it's packaged more or less like a regular app apk,
except that it's built into system image and may be a priveleged app, chances
are that your instrumentation will be targeting the app package (see below
section about manifest) of your component. In this case, your applicaiton
makefile may have its own `LOCAL_CERTIFICATE` setting, and your instrumentation
module should retain the same setting. This is because to target your
instrumentation on the app under test, your test apk and app apk must be signed
with the same certificate.
In other cases, you don't need to have this setting at all: the build system
will simply sign it with a default built-in certificate, based on the build
variant, and it's typically called the `dev-keys`.
```makefile
LOCAL_COMPATIBILITY_SUITE := device-tests
```
This sets up the test to be easily discoverable by the TradeFederation test
harness. Other suites can be added here such as CTS so that this test may be
shared.
```makefile
include $(BUILD_PACKAGE)
```
This includes a core makefile in build system that performs the necessary steps
to generate an apk based on the settings provided by the preceding variables.
The generated apk will be named after `LOCAL_PACKAGE_NAME`, e.g.
`HelloWorldTests.apk`. And if `tests` is used as `LOCAL_MODULE_TAGS` and there
are no other customizations, you should be able to find your test apk in:
* `${OUT}/data/app/<LOCAL_PACKAGE_NAME>/<LOCAL_PACKAGE_NAME>.apk`
e.g. `${OUT}/data/app/HelloWorldTests/HelloWorldTests.apk`
## Manifest file
Just like a regular application, each instrumentation test module needs a
manifest file. If you name the file as `AndroidManifest.xml` and provide it next
to `Android.mk` for your test tmodule, it will get included automatically by the
`BUILD_PACKAGE` core makefile.
Before proceeding further, it's highly recommended to go through the external
[documentation on manifest file](https://developer.android.com/guide/topics/manifest/manifest-intro.html)
first.
This gives an overview of basic components of a manifest file and their
functionalities.
[Latest Manifest File](../../tests/example/instrumentation/AndroidManifest.xml)
A snapshot is included here for convenience:
```xml
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="android.test.example.helloworld"
android:sharedUserId="android.uid.system" >
<uses-sdk android:minSdkVersion="21" android:targetSdkVersion="21" />
<application>
<uses-library android:name="android.test.runner" />
</application>
<instrumentation android:name="android.support.test.runner.AndroidJUnitRunner"
android:targetPackage="android.test.example.helloworld"
android:label="Hello World Test"/>
</manifest>
```
Some select remarks on the manifest file:
```xml
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="android.test.example.helloworld"
```
The `package` attribute is the application package name: this is the unique
identifier that the Android application framework uses to identify an
application (or in this context: your test application). Each user in the system
can only install one application with that package name.
Furthermore, this `package` attribute is the same as what
[`ComponentName#getPackageName()`](https://developer.android.com/reference/android/content/ComponentName.html#getPackageName\(\))
returns, and also the same you would use to interact with various `pm` sub
commands via `adb shell`.
Please also note that although the package name is typically in the same style
as a Java package name, it actually has very few things to do with it. In other
words, your application (or test) package may contain classes with any package
names, though on the other hand, you could opt for simplicity and have your top
level Java package name in your application or test identical to the application
package name.
```xml
android:sharedUserId="android.uid.system"
```
This declares that at installation time, this apk should be granted the same
user id, i.e. runtime identity, as the core platform. Note that this is
dependent on the apk being signed with same certificate as the core platform
(see `LOCAL_CERTIFICATE` in above section), yet they are different concepts:
* some permissions or APIs are signature protected, which requires same
signing certificate
* some permissions or APIs requires the `system` user identity of the caller,
which requires the calling package to share user id with `system`, if it's a
separate package from core platform itself
```xml
<uses-library android:name="android.test.runner" />
```
This is required for all Instrumentation tests since the related classes are
packaged in a separate framework jar library file, therefore requires additional
classpath entries when the test package is invoked by application framework.
```xml
android:targetPackage="android.test.example.helloworld"
```
You might have noticed that the `targetPackage` here is declared the same as the
`package` attribute declared in the `manifest` tag of this file. As mentioned in
[testing basics](../basics/index.md), this category of instrumentation test are
typically intended for testing framework APIs, so it's not very meaningful for
them to have a specific targeted application package, other then itself.
## Test Configuration File
In order to simplify test execution, you also need write a test configuration
file for Android's test harness, [TradeFederation](https://source.android.com/devices/tech/test_infra/tradefed/).
The test configuration can specify special device setup options and default
arguments to supply the test class.
[Latest Test Config File](../../tests/example/instrumentation/AndroidTest.xml)
A snapshot is included here for convenience:
```xml
<configuration description="Runs sample instrumentation test.">
<target_preparer class="com.android.tradefed.targetprep.TestFilePushSetup"/>
<target_preparer class="com.android.tradefed.targetprep.TestAppInstallSetup">
<option name="test-file-name" value="HelloWorldTests.apk"/>
</target_preparer>
<target_preparer class="com.android.tradefed.targetprep.PushFilePreparer"/>
<target_preparer class="com.android.tradefed.targetprep.RunCommandTargetPreparer"/>
<option name="test-suite-tag" value="apct"/>
<option name="test-tag" value="SampleInstrumentationTest"/>
<test class="com.android.tradefed.testtype.AndroidJUnitTest">
<option name="package" value="android.test.example.helloworld"/>
<option name="runner" value="android.support.test.runner.AndroidJUnitRunner"/>
</test>
</configuration>
```
Some select remarks on the test configuration file:
```xml
<target_preparer class="com.android.tradefed.targetprep.TestAppInstallSetup">
<option name="test-file-name" value="HelloWorldTests.apk"/>
</target_preparer>
```
This tells TradeFederation to install the HelloWorldTests.apk onto the target
device using a specified target_preparer. There are many target preparers
available to developers in TradeFederation and these can be used to ensure
the device is setup properly prior to test execution.
```xml
<test class="com.android.tradefed.testtype.AndroidJUnitTest">
<option name="package" value="android.test.example.helloworld"/>
<option name="runner" value="android.support.test.runner.AndroidJUnitRunner"/>
</test>
```
This specifies the TradeFederation test class to use to execute the test and
passes in the package on the device to be executed and the test runner
framework which is JUnit in this case.
Look here for more information on [Test Module Configs](test-config.md)
## JUnit4 Features
Using `android-support-test` library as test runner enables adoptation of new
JUnit4 style test classes, and the sample gerrit change contains some very basic
use of its features.
[Latest source code](../../tests/example/instrumentation/src/android/test/example/helloworld/HelloWorldTest.java)
While testing patterns are usually specific to component teams, there are some
generally useful usage patterns.
```java
@RunWith(JUnit4.class)
public class HelloWorldTest {
```
A significant difference in JUnit4 is that tests are no longer required to
inherit from a common base test class; instead, you write tests in plain Java
classes and use annotation to indicate certain test setup and constraints. In
this example, we are instructing that this class should be run as a JUnit4 test.
```java
@BeforeClass
public static void beforeClass() {
...
@AfterClass
public static void afterClass() {
...
@Before
public void before() {
...
@After
public void after() {
...
@Test
@SmallTest
public void testHelloWorld() {
...
```
The `@Before` and `@After` annotations are used on methods by JUnit4 to perform
pre test setup and post test teardown. Similarly, the `@BeforeClass` and
`@AfterClass` annotations are used on methods by JUnit4 to perform setup before
executing all tests in a test class, and teardown afterwards. Note that the
class-scope setup and teardown methods must be static. As for the test methods,
unlike in earlier version of JUnit, they no longer need to start the method name
with `test`, instead, each of them must be annotated with `@Test`. As usual,
test methods must be public, declare no return value, take no parameters, and
may throw exceptions.
**Important**: the test methods themselves are annotated with `@Test`
annotation; and note that for tests to be executed via APCT, they must be
annotated with test sizes: the example annotated method `testHelloWorld` as
`@SmallTest`. The annotation may be applied at method scope, or class scope.
## Accessing `Instrumentation`
Although not covered in the basic hello world example, it's fairly common for an
Android test to require access `Instrumentation` instance: this is the core API
interface that provides access to application contexts, activity lifecycle
related test APIs and more.
Because the JUnit4 tests no longer require a common base class, it's no longer
necessary to obtain `Instrumentation` instance via
`InstrumentationTestCase#getInstrumentation()`, instead, the new test runner
manages it via [`InstrumentationRegistry`](https://developer.android.com/reference/android/support/test/InstrumentationRegistry.html)
where contextual and environmental setup created by instrumentation framework is
stored.
To access the instance of `Instrumentation` class, simply call static method
`getInstrumentation()` on `InstrumentationRegistry` class:
```java
Instrumentation instrumentation = InstrumentationRegistry.getInstrumentation()
```
## Build & Test Locally:
Follow these [Instructions](instrumentation.md)

94
android/platform_testing/docs/development/instrumentation.md Normal file
View file

@ -0,0 +1,94 @@
# Instrumentation Tests
1. Below are common destinations for hermetic tests against framework services:
```
frameworks/base/core/tests/coretests
frameworks/base/services/tests/servicestests
```
If you are adding a brand new instrumentation module for your component, see
* [Self-Instrumenting Tests: A Complete Example](instr-self-e2e.md)
* [Instrumentation Targeting an Application: A Complete Example]
(instr-app-e2e.md)
1. Following the existing convention if you are adding tests into one of the
locations above. If you are setting up a new test module, please follow the
setup of `AndroidManifest.xml` and `Android.mk` in one of the locations
above
1. See https://android.googlesource.com/platform/frameworks/base.git/+/master/core/tests/coretests/ for an example
1. Note: do not forget to mark your test as `@SmallTest`, `@MediumTest` or
`@LargeTest`
1. Build the test module with make, e.g.:
```
make FrameworksCoreTests -j
```
1. Automatic installation and run with the TradeFederation test harness:
```
make tradefed-all -j
tradefed.sh run template/local_min --template:map test=FrameworksCoreTests
```
1. Manually Install and Run:
1. Install the generated apk:
```
adb install -r ${OUT}/data/app/FrameworksCoreTests/FrameworksCoreTests.apk
```
Tip: you use `adb shell pm list instrumentation` to find the
instrumentations inside the apk just installed
1. Run the tests with various options:
1. all tests in the apk
```
adb shell am instrument -w com.android.frameworks.coretests\
/android.support.test.runner.AndroidJUnitRunner
```
1. all tests under a specific Java package
```
adb shell am instrument -w -e package android.animation \
com.android.frameworks.coretests\
/android.support.test.runner.AndroidJUnitRunner
```
1. all tests under a specific class
```
adb shell am instrument -w -e class \
android.animation.AnimatorSetEventsTest \
com.android.frameworks.coretests\
/android.support.test.runner.AndroidJUnitRunner
```
1. a specific test method
```
adb shell am instrument -w -e class \
android.animation.AnimatorSetEventsTest#testCancel \
com.android.frameworks.coretests\
/android.support.test.runner.AndroidJUnitRunner
```
Your test can make an explicit assertion on pass or fail using `JUnit` APIs; in
addition, any uncaught exceptions will also cause a functional failure.
To emit performance metrics, your test code can call
[`Instrumentation#sendStatus`](http://developer.android.com/reference/android/app/Instrumentation.html#sendStatus\(int, android.os.Bundle\))
to send out a list of key-value pairs. It's important to note that:
1. metrics can be integer or floating point
1. any non-numerical values will be discarded
1. your test apk can be either functional tests or metrics tests, however
mixing both are not currently supported

34
android/platform_testing/docs/development/metrics.md Normal file
View file

@ -0,0 +1,34 @@
# Native Metric Tests
As mentioned earlier, native metric tests are typically used for exercising HAL
or interacting directly with lower level system services, and to leverage
continuous testing service, native metric tests should be built with
[google-benchmark](https://github.com/google/benchmark) framework.
Here are some general instructions:
1. See sample native test module setup at: `bionic/benchmarks/bionic-benchmarks`
1. Test module makefile should use `BUILD_NATIVE_BENCHMARK` build rule so that
google-benchmark dependencies are included automatically
1. Build the test module with make:
```shell
make -j40 bionic-benchmarks
```
1. Automatic installation and run with the TradeFederation test harness:
```
make tradefed-all -j
tradefed.sh run template/local_min --template:map test=bionic-benchmarks
1. Manually Install and Run:
1. Push the generated test binary onto device:
```shell
adb push ${OUT}/data/benchmarktest/bionic-benchmarks/bionic-benchmarks32 \
/data/benchmarktest/bionic-benchmarks/bionic-benchmarks32
```
1. Execute the test by invoking test binary on device:
```shell
adb shell /data/benchmarktest/bionic-benchmarks/bionic-benchmarks32
```

269
android/platform_testing/docs/development/native-func-e2e.md Normal file
View file

@ -0,0 +1,269 @@
# Adding a New Native Test: A Complete Example
[TOC]
If you are new to Android platform development, you might find this complete
example of adding a brand new native test from scratch useful to demonstrate the
typical workflow involved.
Note that this guide assumes that you already have some knowledge in the
platform source tree workflow. If not, please refer to
https://source.android.com/source/requirements.
In addition, if you are also unfamiliar with the gtest framework for C++, please
check out its project page first:
* https://github.com/google/googletest
This guide uses the follow test to serve as an sample:
* [Hello World Native Test](../../tests/example/native)
It's recommended to browse through the code first to get a rough impression
before proceeding.
## Deciding on a Source Location
Typically your team will already have an established pattern of places to check
in code, and places to add tests. Most team owns a single git repository, or
share one with other teams but have a dedicated sub directory that contains
component source code.
Assuming the root location for your component source is at `<component source
root>`, most components have `src` and `tests` folders under it, and some
additional files such as `Android.mk` (or broken up into additional `.mk`
files).
Since you are adding a brand new test, you'll probably need to create the
`tests` directory next to your component `src`, and populate it with content.
In some cases, your team might have further directory structures under `tests`
due to the need to package different suites of tests into individual binaries.
And in this case, you'll need to create a new sub directory under `tests`.
To illustrate, here's a typical directory outline for components with a single
`tests` folder:
```
\
<component source root>
\-- Android.mk (component makefile)
\-- AndroidTest.mk (test config file)
\-- src (component source)
| \-- foo.cpp
| \-- ...
\-- tests (test source root)
\-- Android.mk (test makefile)
\-- src (test source)
\-- foo_test.cpp
\-- ...
```
and here's a typical directory outline for components with multiple test source
directories:
```
\
<component source root>
\-- Android.mk (component makefile)
\-- AndroidTest.mk (test config file)
\-- src (component source)
| \-- foo.cpp
| \-- ...
\-- tests (test source root)
\-- Android.mk (test makefile)
\-- testFoo (sub test source root)
| \-- Android.mk (sub test makefile)
| \-- src (sub test source)
| \-- test_foo.cpp
| \-- ...
\-- testBar
| \-- Android.mk
| \-- src
| \-- test_bar.cpp
| \-- ...
\-- ...
```
Regardless of the structure, you'll end up populating the `tests` directory or
the newly created sub directory with files similar to what's in `native`
directory in the sample gerrit change. The sections below will explain in
further details of each file.
## Makefile
Each new test module must have a makefile to direct the build system with module
metadata, compile time dependencies and packaging instructions.
[Latest version of the makefile](../../tests/example/native/Android.mk)
A snapshot is included here for convenience:
```makefile
LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_SRC_FILES := \
HelloWorldTest.cpp
LOCAL_MODULE := hello_world_test
LOCAL_MODULE_TAGS := tests
LOCAL_COMPATIBILITY_SUITE := device-tests
include $(BUILD_NATIVE_TEST)
```
Some select remarks on the makefile:
```makefile
LOCAL_MODULE := hello_world_test
```
This setting declares the module name, which must be unique in the entire build
tree. It will also be used as the name as the binary executable of your test, as
well as a make target name, so that you can use `make [options] <LOCAL_MODULE>`
to build your test binary and all its dependencies.
```makefile
LOCAL_MODULE_TAGS := tests
```
This setting declares the module as a test module, which will instruct the build
system to generate the native test binaries under "data" output directory, so
that they can be packaged into test artifact file bundle.
```makefile
LOCAL_COMPATIBILITY_SUITE := device-tests
```
This line builds the testcase as part of the device-tests suite, which is
meant to target a specific device and not a general ABI.
```makefile
include $(BUILD_NATIVE_TEST)
```
This includes a core makefile in build system that performs the necessary steps
to compile your test, together with gtest framework under `external/gtest`, into
a native test binary. The generated binary will have the same name as
`LOCAL_MODULE`. And if `tests` is used as `LOCAL_MODULE_TAGS` and there are no
other customizations, you should be able to find your test binary in:
* `${OUT}/data/nativetest[64]/<LOCAL_MODULE>/<LOCAL_MODULE>`
e.g. `${OUT}/data/nativetest[64]/hello_world_test/hello_world_test`
And you will also find it:
* ${OUT}/target/product/<target>/testcases/<LOCAL_MODULE>/<arch>/<LOCAL_MODULE>
e.g. ${OUT}/target/product/arm64-generic/testcases/hello_world_test/arm/hello_world_test
& ${OUT}/target/product/arm64-generic/testcases/hello_world_test/arm64/hello_world_test
Note: if the native ABI type of device is 64bit, such as angler, bullhead etc,
the directory name will be suffixed with `64`.
Please also note that currently the native tests in APCT does not support use of
dynamically linked libraries, which means that the dependencies needs to be
statically linked into the test binary.
## Source code
[Latest source code](../../tests/example/native/HelloWorldTest.cpp)
Annotated source code is listed below:
```c++
#include <gtest/gtest.h>
```
Header file include for gtest. Note that the include file dependency is
automatically resolved by using `BUILD_NATIVE_TEST` in the makefile
```c++
#include <stdio.h>
TEST(HelloWorldTest, PrintHelloWorld) {
printf("Hello, World!");
}
```
gtests are written by using `TEST` macro: the first parameter is the test case
name, and the second is test name; together with test binary name, they form the
hierarchy below when visualized in result dashboard:
```
<test binary 1>
| \-- <test case 1>
| | \-- <test 1>
| | \-- <test 2>
| | \-- ...
| \-- <test case 2>
| | \-- <test 1>
| | \-- ...
| \-- ...
<test binary 2>
|
...
```
For more information on writing tests with gtest, see its documentation:
* https://github.com/google/googletest/blob/master/googletest/docs/Primer.md
## Test Config
In order to simplify test execution, you also need write a test configuration
file for Android's test harness, [TradeFederation](https://source.android.com/devices/tech/test_infra/tradefed/).
The test configuration can specify special device setup options and default
arguments to supply the test class.
[LATEST TEST CONFIG](../../tests/example/native/AndroidTest.xml)
A snapshot is included here for convenience:
```xml
<configuration description="Config for APCT native hello world test cases">
<target_preparer class="com.android.tradefed.targetprep.PushFilePreparer">
<option name="cleanup" value="true" />
<option name="push" value="hello_world_test->/data/local/tmp/hello_world_test" />
</target_preparer>
<test class="com.android.tradefed.testtype.GTest" >
<option name="native-test-device-path" value="/data/local/tmp" />
<option name="module-name" value="hello_world_test" />
<option name="runtime-hint" value="8m" />
</test>
</configuration>
```
Some select remarks on the test configuration file:
```xml
<target_preparer class="com.android.tradefed.targetprep.PushFilePreparer">
<option name="cleanup" value="true" />
<option name="push" value="hello_world_test->/data/local/tmp/hello_world_test" />
</target_preparer>
```
This tells TradeFederation to install the hello_world_test binary onto the target
device using a specified target_preparer. There are many target preparers
available to developers in TradeFederation and these can be used to ensure
the device is setup properly prior to test execution.
```xml
<test class="com.android.tradefed.testtype.GTest" >
<option name="native-test-device-path" value="/data/local/tmp" />
<option name="module-name" value="hello_world_test" />
<option name="runtime-hint" value="8m" />
</test>
```
This specifies the TradeFederation test class to use to execute the test and
passes in the native test location that it was installed.
Look here for more information on [Test Module Configs](../test-config.md)
## Build & Test Locally
Follow these [Instructions](../native.md) to build and execute your test

43
android/platform_testing/docs/development/native.md Normal file
View file

@ -0,0 +1,43 @@
# Native Tests
As mentioned earlier, native tests are typically used for exercising HAL or
interacting directly with lower level system services, and to leverage
continuous testing service, native tests should be built with
[gtest](https://github.com/google/googletests) framework.
Here are some general instructions:
1. See sample native test module setup at: `libs/hwui/unit_tests`
1. Test module makefile should use `BUILD_NATIVE_TEST` build rule so that
gtest dependencies are included automatically
1. Write a [test config](../test-config.md)
1. Build the test module with `mmm` or `mma` (depends on if it's an
incremental or full build), e.g.:
```shell
make hwui_unit_tests -j
```
1. Automatic installation and run with the TradeFederation test harness:
```
make tradefed-all -j
tradefed.sh run template/local_min --template:map test=hwui_unit_tests
```
1. Manually Install and Run:
1. Push the generated test binary onto device:
```shell
adb push ${OUT}/data/nativetest/hwui_unit_tests/hwui_unit_tests \
/data/nativetest/hwui_unit_tests/hwui_unit_tests
```
1. Execute the test by invoking test binary on device:
```shell
adb shell /data/nativetest/hwui_unit_tests/hwui_unit_tests
```
This launches the native test. You can also add `--help` parameter to your test
binary to find out more about the different ways to customize test execution.
You can also check out the gtest advanced guide for more parameters usage:
* https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md

155
android/platform_testing/docs/development/test-config.md Normal file
View file

@ -0,0 +1,155 @@
## Test Module Config
Some test modules may require customized setup and tear down steps that cannot
be performed within test case itself. Typical examples may include:
* install other apks (in addition to the test apk)
* push some files to the device
* run commands (e.g. adb shell pm ...)
In the past, component teams usually resort to writing a host side test to
perform such tasks, which requires understanding of TradeFederation harness
and typically increases the complexity of a test module .
Borrowing from CTS, we introduced the concept of test module config to support
such tasks, the common tasks list above can be achieved by just a few lines of
config. For maximum flexibility, you can even implement your own target
preparer, as defined by [ITargetPreparer]
(https://source.android.com/reference/com/android/tradefed/targetprep/ITargetPreparer.html)
or [ITargetCleaner]
(https://source.android.com/reference/com/android/tradefed/targetprep/ITargetCleaner.html),
and configure them to use in your own test module config.
A test module config for a test module is a required XML file added to the top
level module source folder, named ‘AndroidTest.xml’. The XML follows the format
of a configuration file used by TradeFederation test automation harness.
Currently the main tags handled via the test module configs are the “target_preparer” and
"test" tags.
## Target Preparers
A “target_preparer” tag, as the name suggests, defines a target preparer
(see [ITargetPreparer](https://source.android.com/reference/com/android/tradefed/targetprep/ITargetPreparer.html))
that offers a setup method, which gets called before the test module is executed
for testing; and if the class referenced in the “target_preparer” tag also
implements
[ITargetCleaner](https://source.android.com/reference/com/android/tradefed/targetprep/ITargetCleaner.html),
its teardown method will be invoked after the test module has finished.
To use the built-in common module config, add a new file ‘AndroidTest.xml’ at
the top level folder for your test module, and populate it with the following
content:
```xml
<?xml version="1.0" encoding="utf-8"?>
<!-- [insert standard AOSP copyright here] -->
<configuration description="Test module config for Foo">
<!-- insert options here -->
</configuration>
```
As an example, we can add the following option tags (at the “insert” comment
above):
```xml
<target_preparer class="com.android.tradefed.targetprep.RunCommandTargetPreparer">
<option name="run-command" value="settings put secure accessibility_enabled 1" />
<option name="teardown-command" value="settings put secure accessibility_enabled 0" />
</target_preparer>
```
The options will configure the test harness to:
1. before test module is invoked, execute shell command “settings put secure
accessibility_enabled 1” on device
2. after test module is finished, execute shell command “settings put secure
accessibility_enabled 0”
In this particular example, accessibility is enabled/disabled before/after the
test module execution, respectively. With a simple example demonstrated, it’s
necessary to cover more details on how the “option” tag is used. As shown above,
the tag can have two attributes: name, value. The name attribute indicated the
name of the option, and is further broken down into two parts separated by a
colon: short name for the preparer, and the actual option name offered by the
preparer. The exact purpose of value field is dependent on how preparer defined
the option: it can be a string, a number, a boolean, or even a file path etc. In
the example above, name “run-command:run-command” means that we are setting
value for the option “run-command” defined by a target preparer with short name
“run-command”; and name “run-command:teardown-command” means that we are setting
value for the option “teardown-command” also defined by the same target preparer
with short name “run-command”. Here's a summary of the 3 common target
preparers:
* class name: [PushFilePreparer](https://android.googlesource.com/platform/tools/tradefederation/+/master/src/com/android/tradefed/targetprep/PushFilePreparer.java)
* **short name**: push-file
* **function**: pushes arbitrary files under test case folder into
destination on device
* **notes**:
* this preparer can push from folder to folder, or file to file; that
is, you cannot push a file under a folder on device: you must
specify the destination filename under that folder as well
* **options**:
* **push:** A push-spec, formatted as
'`/path/to/srcfile.txt->/path/to/destfile.txt`' or
'`/path/to/srcfile.txt->/path/to/destdir/`'. May be repeated
This path may be relative to the test module directory or the out
directory itself.
* **post-push: **A command to run on the device (with \``adb shell
<your command>`\`) after all pushes have been attempted. Typical use
case would be using chmod for permissions
* class name: [InstallApkSetup](https://android.googlesource.com/platform/tools/tradefederation/+/master/src/com/android/tradefed/targetprep/InstallApkSetup.java)
* **short name:**install-apk
* **function:** pushes arbitrary apk files under into destination on
device
* **options:**
* **test-file-name:** the name of the apk to be installed on to
device.
* **install-arg:** Additional arguments to be passed to the pm install
command, including leading dash, e.g. “-d". May be repeated
* class name: [RunCommandTargetPreparer](https://android.googlesource.com/platform/tools/tradefederation/+/master/src/com/android/tradefed/targetprep/RunCommandTargetPreparer.java)
* **short name:** run-command
* **function:** executes arbitrary shell commands before or after test
module execution
* **options:**
* **run-command:**adb shell command to run. May be repeated
* **teardown-command:**adb shell command to run during teardown phase.
May be repeated
## Test Class
A test class is the TradeFederation class to use to execute the test.
```xml
<test class="com.android.tradefed.testtype.AndroidJUnitTest">
<option name="package" value="android.test.example.helloworld"/>
<option name="runner" value="android.support.test.runner.AndroidJUnitRunner"/>
</test>
```
Here are 3 common test classes:
* class name: [GTest](https://android.googlesource.com/platform/tools/tradefederation/+/master/src/com/android/tradefed/testtype/GTest.java)
* **short name:** gtest
* **function:** A Test that runs a native test package on given device.
* **options:**
* **native-test-device-path:**The path on the device where native tests are located.
* class name: [InstrumentationTest](https://android.googlesource.com/platform/tools/tradefederation/+/master/src/com/android/tradefed/testtype/InstrumentationTest.java)
* **short name:** instrumentation
* **function:** A Test that runs an instrumentation test package on given device
* **options:**
* **package:**The manifest package name of the Android test application to run.
* **class:**The test class name to run.
* **method:**The test method name to run.
* class name: [AndroidJUnitTest](https://android.googlesource.com/platform/tools/tradefederation/+/master/src/com/android/tradefed/testtype/AndroidJUnitTest.java)
* **function:** A Test that runs an instrumentation test package on given
device using the android.support.test.runner.AndroidJUnitRunner
This is the main way to execute an instrumentation test.