978 lines
20 KiB
Text
978 lines
20 KiB
Text
#LyX 1.1 created this file. For more info see http://www.lyx.org/
|
|
\lyxformat 2.16
|
|
\textclass docbook
|
|
\begin_preamble
|
|
<!entity header system "header.sgml">
|
|
\end_preamble
|
|
\language default
|
|
\inputencoding default
|
|
\fontscheme default
|
|
\graphics default
|
|
\paperfontsize default
|
|
\spacing single
|
|
\papersize Default
|
|
\paperpackage a4
|
|
\use_geometry 0
|
|
\use_amsmath 0
|
|
\paperorientation portrait
|
|
\secnumdepth 3
|
|
\tocdepth 3
|
|
\paragraph_separation indent
|
|
\defskip medskip
|
|
\quotes_language english
|
|
\quotes_times 2
|
|
\papercolumns 1
|
|
\papersides 1
|
|
\paperpagestyle default
|
|
|
|
\layout Title
|
|
\added_space_top vfill \added_space_bottom vfill
|
|
Linux Test Project HOWTO
|
|
\layout Date
|
|
|
|
10 October 2000
|
|
\layout Author
|
|
|
|
Nate Straz
|
|
\layout Abstract
|
|
|
|
This document explains some of the more in depth topics of the Linux Test
|
|
Project and related testing issues.
|
|
It does not cover basic installation procedures.
|
|
See the INSTALL and README files in the tarball for that information.
|
|
\layout Section
|
|
|
|
Preface
|
|
\layout Standard
|
|
|
|
This document was written to help bring the community up to speed on the
|
|
ins and outs of the Linux Test Project.
|
|
\layout Subsection
|
|
|
|
Copyright
|
|
\layout Standard
|
|
|
|
Copyright (c) 2000 by SGI, Inc.
|
|
|
|
\layout Standard
|
|
|
|
Please freely copy and distribute (sell or give away) this document in any
|
|
format.
|
|
It's requested that corrections and/or comments be fowarded to the document
|
|
maintainer.
|
|
You may create a derivative work and distribute it provided that you:
|
|
\layout Itemize
|
|
|
|
Send your derivative work (in the most suitable format such as sgml) to
|
|
the LDP (Linux Documentation Project) or the like for posting on the Internet.
|
|
If not the LDP, then let the LDP know where it is available.
|
|
|
|
\layout Itemize
|
|
|
|
License the derivative work with this same license or use GPL.
|
|
Include a copyright notice and at least a pointer to the license used.
|
|
|
|
\layout Itemize
|
|
|
|
Give due credit to previous authors and major contributors.
|
|
|
|
\layout Standard
|
|
|
|
If you're considering making a derived work other than a translation, it's
|
|
requested that you discuss your plans with the current maintainer.
|
|
|
|
\layout Subsection
|
|
|
|
Disclaimer
|
|
\layout Standard
|
|
|
|
Use the information in this document at your own risk.
|
|
I disavow any potential liability for the contents of this document.
|
|
Use of the concepts, examples, and/or other content of this document is
|
|
entirely at your own risk.
|
|
|
|
\layout Standard
|
|
|
|
All copyrights are owned by their owners, unless specifically noted otherwise.
|
|
Use of a term in this document should not be regarded as affecting the
|
|
validity of any trademark or service mark.
|
|
|
|
\layout Standard
|
|
|
|
Naming of particular products or brands should not be seen as endorsements.
|
|
|
|
\layout Standard
|
|
|
|
You are strongly recommended to take a backup of your system before major
|
|
installation and backups at regular intervals.
|
|
|
|
\layout Section
|
|
|
|
Introduction
|
|
\layout Subsection
|
|
|
|
What is the Linux Test Project?
|
|
\layout Standard
|
|
|
|
The Linux Test Project (LTP) is an effort to create a set of tools and tests
|
|
to verify the functionality and stability of the Linux kernel.
|
|
We hope this will support Linux development by making unit testing more
|
|
complete and minimizing user impact by building a barrier to keep bugs
|
|
from making it to the user.
|
|
|
|
\layout Subsection
|
|
|
|
What is wrong with the current testing model?
|
|
\layout Standard
|
|
|
|
The Linux development community utilizes two important (some out argue most
|
|
important) testing techniques in its normal operations: Design and Code
|
|
Inspections.
|
|
The intent of LTP is to support this by giving developers an ever growing
|
|
set of tools to help identify any operational problems in their code that
|
|
may be missed by human review.
|
|
One of the toughest categories of problems to catch with inspection is
|
|
that of interaction of features.
|
|
With a continuously improving set of tests and tools, developers can get
|
|
an indication of whether their changes may have broken some other functionality.
|
|
|
|
\layout Standard
|
|
|
|
There is no such thing as a perfect test base.
|
|
It is only useful it if keeps up with new and changing functionality,
|
|
and if it actually gets used.
|
|
|
|
\layout Subsection
|
|
|
|
Are you doing benchmarking?
|
|
\layout Standard
|
|
|
|
Not at this time.
|
|
We are more interested in functional, regression, and stress testing the
|
|
Linux kernel.
|
|
Benchmarking may be workable to compare the performance among kernel versions.
|
|
|
|
\layout Subsection
|
|
|
|
Are you doing standards testing?
|
|
\layout Standard
|
|
|
|
No, we are leaving that to the Linux Standards Base (LSB).
|
|
See the Linux Standards Base
|
|
\begin_inset LatexCommand \htmlurl[web site]{http://www.linuxbase.org/}
|
|
|
|
\end_inset
|
|
|
|
for more information.
|
|
|
|
\layout Section
|
|
|
|
Structure
|
|
\layout Standard
|
|
|
|
The basic building block of the test project is a
|
|
\series bold
|
|
test case
|
|
\series default
|
|
that consists of a single action and a verification that the action worked.
|
|
The result of the test case is usually restricted to PASS/FAIL.
|
|
|
|
\layout Standard
|
|
|
|
A
|
|
\series bold
|
|
test program
|
|
\series default
|
|
is a runnable program that contains one or more test cases.
|
|
Test programs often understand command line options which alter their behavior.
|
|
The options could determine the amount of memory tested, the location of
|
|
temporary files, the type of network packet used, or any other useful parameter.
|
|
\layout Standard
|
|
|
|
|
|
\series bold
|
|
Test tags
|
|
\series default
|
|
are used to pair a unique identifier with a test program and a set of command
|
|
line options.
|
|
Test tags are the basis for test suites.
|
|
\layout Section
|
|
|
|
Writing Tests
|
|
\layout Standard
|
|
|
|
Writing a test case is a lot easier than most people think.
|
|
Any code that you write to examine how a part of the kernel works can
|
|
be adapted into a test case.
|
|
All that is needed is a way to report the result of the action to the
|
|
rest of the world.
|
|
There are several ways of doing this, some more involved than others.
|
|
|
|
\layout Subsection
|
|
|
|
Exit Style Tests
|
|
\layout Standard
|
|
|
|
Probably the simplest way of reporting the results of a test case is the
|
|
exit status of your program.
|
|
If your test program encounters unexpected or incorrect results, exit
|
|
the test program with a non-zero exit status, i.e.
|
|
|
|
\family typewriter
|
|
exit(1)
|
|
\family default
|
|
.
|
|
Conversely, if your program completes as expected, return a zero exit status,
|
|
i.e.
|
|
|
|
\family typewriter
|
|
exit(0)
|
|
\family default
|
|
.
|
|
Any test driver should be able to handle this type of error reporting.
|
|
If a test program has multiple test cases you won't know which test case
|
|
failed, but you will know the program that failed.
|
|
|
|
\layout Subsection
|
|
|
|
Formatted Output Tests
|
|
\layout Standard
|
|
|
|
The next easiest way of reporting the results is to write the results of
|
|
each test case to standard output.
|
|
This allows for the testing results to be more understandable to both the
|
|
tester and the analysis tools.
|
|
When the results are written in a standard way, tools can be used to analyze
|
|
the results.
|
|
|
|
\layout Section
|
|
|
|
Testing Tools
|
|
\layout Standard
|
|
|
|
The Linux Test Project has not yet decided on a "final" test harness.
|
|
We have provided a simple solution with
|
|
\family typewriter
|
|
ltp-pan
|
|
\family default
|
|
to make due until a complete solution has been found/created that compliments
|
|
the Linux kernel development process.
|
|
Several people have said we should use such and such a test harness.
|
|
Until we find we need a large complex test harness, we will apply the KISS
|
|
concept.
|
|
|
|
\layout Subsection
|
|
|
|
Ltp-pan
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
ltp-pan
|
|
\family default
|
|
is a simple test driver with the ability to keep track of orphaned processes
|
|
and capture test output.
|
|
It works by reading a list of test tags and command lines and runs them.
|
|
By default ltp-pan will select a command randomly from the list of test tags,
|
|
wait for it to finish.
|
|
Through command line options you can run through the entire list sequentially,
|
|
run n tests, keep n test running at all times, and buffer test output.
|
|
Ltp-pan can be nested to create very complex test environments.
|
|
\layout Standard
|
|
|
|
Ltp-pan uses an
|
|
\emph on
|
|
active file
|
|
\emph default
|
|
, also called a
|
|
\emph on
|
|
zoo file
|
|
\emph default
|
|
to keep track of which tests are currently running.
|
|
This file holds the pid, tag, and a portion of the command line.
|
|
When you start ltp-pan it becomes a test tag in itself, thus it requires a
|
|
name for itself.
|
|
Ltp-pan updates the active file to show which test tags are currently running.
|
|
When a test tag exits, ltp-pan will overwrite the first character with a '#'.
|
|
The active file can be shared between multiple instances of ltp-pan so you
|
|
know which tests were running when the system crashes by looking at one
|
|
file.
|
|
|
|
\layout Standard
|
|
|
|
A
|
|
\emph on
|
|
ltp-pan file
|
|
\emph default
|
|
contains a list of test tags for ltp-pan to run.
|
|
The format of a ltp-pan file is as follows:
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
testtag testprogram -o one -p two other command line options
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# This is a comment.
|
|
It is a good idea to describe the test
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# tags in your ltp-pan file.
|
|
Tests programs can have different
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# behaviors depending on the command line options so it is
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# helpful to describe what each test tag is meant to verify or # provoke.
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# Some more test cases
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
mm01 mmap001 -m 10000
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# 40 Mb mmap() test.
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# Creates a 10000 page mmap, touches all of the map, sync's
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# it, and munmap()s it.
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
mm03 mmap001 -i 0 -I 1 -m 100
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# repetitive mmapping test.
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# Creates a one page map repetitively for one minute.
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
dup02 dup02
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# Negative test for dup(2) with bad fd
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
kill09 kill09
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# Basic test for kill(2)
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
fs-suite01 ltp-pan -e -a fs-suite01.zoo -n fs-suite01 -f runtest/fs
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
# run the entire set of file system tests
|
|
\layout Standard
|
|
|
|
The test tags are simple identifiers, no spaces are allowed.
|
|
The test of the line is the program to run, which is done using execvp(3).
|
|
Lines starting with '#' are comments and ignored by ltp-pan.
|
|
It is a good practice to include descriptions with your test tags so you
|
|
can have a reminder what a certain obscure test tag tries to do.
|
|
\layout Subsubsection
|
|
|
|
Examples
|
|
\layout Standard
|
|
|
|
The most basic way to run ltp-pan is by passing the test program and parameters
|
|
on the command line.
|
|
This will run the single program once and wrap the output.
|
|
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
$ ltp-pan -a ltp.zoo -n tutor sleep 4
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<test_start>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
tag=cmdln stime=971450564
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
cmdline="sleep 4"
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
contacts=""
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
analysis=exit
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
initiation_status="ok"
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<test_output>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<execution_status>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
duration=103341903 termination_type=exited termination_id=0 corefile=no
|
|
cutime=0 cstime=0
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<test_end>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
$ cat ltp.zoo
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
#9357,tutor,pan/ltp-pan -a ltp.zoo -n tutor sleep 4
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
#9358,cmdln,sleep 4
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
$
|
|
\layout Paragraph
|
|
|
|
How it works
|
|
\layout Standard
|
|
|
|
This example shows the two parameters that are always required by ltp-pan, the
|
|
active file and a test tag for ltp-pan.
|
|
The
|
|
\begin_inset Quotes eld
|
|
\end_inset
|
|
|
|
sleep 4
|
|
\begin_inset Quotes erd
|
|
\end_inset
|
|
|
|
on the end of the command line is a test program and parameters that ltp-pan
|
|
should run.
|
|
This test is given the tag
|
|
\begin_inset Quotes eld
|
|
\end_inset
|
|
|
|
cmdln.
|
|
\begin_inset Quotes erd
|
|
\end_inset
|
|
|
|
Ltp-pan will run one test randomly, which ends up being cmdln since it is the
|
|
only test that we told ltp-pan about.
|
|
|
|
\layout Standard
|
|
|
|
In the active file,
|
|
\family typewriter
|
|
ltp.zoo
|
|
\family default
|
|
, ltp-pan writes the pid, test tag, and part of the command line for the currently
|
|
running tests.
|
|
The command lines are truncated so each line will fit on an 80 column display.
|
|
When a test tag finishes, ltp-pan will place a '#' at the beginning of the
|
|
line to mark it as available.
|
|
Here you can see that cmdln and tutor, the name we gave ltp-pan, ran to completion.
|
|
If the computer hangs, you can read this file to see which test programs
|
|
were running.
|
|
\layout Standard
|
|
|
|
We have run one test once.
|
|
Let's do something a little more exciting.
|
|
Let's run one test several times, at the same time.
|
|
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
$ ltp-pan -a ltp.zoo -n tutor -x 3 -s 3 -O /tmp sleep 1
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<test_start>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
tag=cmdln stime=971465653
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
cmdline="sleep 1"
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
contacts=""
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
analysis=exit
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
initiation_status="ok"
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<test_output>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<execution_status>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
duration=103326814 termination_type=exited termination_id=0 corefile=no
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
cutime=1 cstime=0
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<test_end>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<test_start>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
tag=cmdln stime=971465653
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
cmdline="sleep 1"
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
contacts=""
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
analysis=exit
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
initiation_status="ok"
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<test_output>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<execution_status>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
duration=103326814 termination_type=exited termination_id=0 corefile=no
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
cutime=0 cstime=1
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<test_end>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<test_start>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
tag=cmdln stime=971465653
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
cmdline="sleep 1"
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
contacts=""
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
analysis=exit
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
initiation_status="ok"
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<test_output>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<execution_status>>>
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
duration=103326814 termination_type=exited termination_id=0 corefile=no
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
cutime=0 cstime=0
|
|
\layout Code
|
|
|
|
|
|
\latex no_latex
|
|
<<<test_end>>>
|
|
\layout Paragraph
|
|
|
|
How it works
|
|
\layout Standard
|
|
|
|
In this example we run another fake test from the command line, but we run
|
|
it three times (-s 3) and keep three test tags active at the same time
|
|
(-x 3).
|
|
The -O parameter is a directory where temporary files can be created to
|
|
buffer the output of each test tag.
|
|
You can see in the output that cmdln ran three times.
|
|
If the -O option were omitted, your test output would be mixed, making
|
|
it almost worthless.
|
|
|
|
\layout Itemize
|
|
|
|
Using a ltp-pan file to run multiple tests
|
|
\layout Itemize
|
|
|
|
Nesting ltp-pan
|
|
\layout Standard
|
|
|
|
For more information on ltp-pan see the man page
|
|
\family typewriter
|
|
doc/man1/ltp-pan.1
|
|
\family default
|
|
.
|
|
\layout Subsection
|
|
|
|
Scanner
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
Ltp-scanner
|
|
\family default
|
|
is a results analysis tool that understands the
|
|
\emph on
|
|
rts
|
|
\emph default
|
|
style output which
|
|
\family typewriter
|
|
ltp-pan
|
|
\family default
|
|
generates by default.
|
|
It will produce a table summarizing which tests passed and which failed.
|
|
|
|
\layout Subsection
|
|
|
|
The Quick-hitter Package
|
|
\layout Standard
|
|
|
|
Many of the tests released use the Quick-hitter test package to perform
|
|
tasks like create and move to a temporary directory, handle some common
|
|
command line parameters, loop, run in parallel, handle signals, and clean
|
|
up.
|
|
|
|
\layout Standard
|
|
|
|
There is an example test case,
|
|
\family typewriter
|
|
doc/examples/quickhit.c
|
|
\family default
|
|
, which shows how the quick-hitter package can be used.
|
|
The file is meant to be a supplement to the documentation, not a working
|
|
test case.
|
|
Use any of the tests in
|
|
\family typewriter
|
|
tests/
|
|
\family default
|
|
as a template.
|
|
\layout Section
|
|
|
|
To Do
|
|
\layout Standard
|
|
|
|
There are a lot of things that still need to be done to make this a complete
|
|
kernel testing system.
|
|
The following sections will discuss some of the to do items in detail.
|
|
|
|
\layout Subsection
|
|
|
|
Configuration Analysis
|
|
\layout Standard
|
|
|
|
While the number of configuration options for the Linux kernel is seen as
|
|
a strength to developers and users alike, it is a curse to testers.
|
|
To create a powerful automated testing system, we need to be able to determine
|
|
what the configuration on the booted box is and then determine which tests
|
|
should be run on that box.
|
|
|
|
\layout Standard
|
|
|
|
The Linux kernel has hundreds of configuration options that can be set to
|
|
compile the kernel.
|
|
There are more options that can be set when you boot the kernel and while
|
|
it is running.
|
|
There are also many patches that can be applied to the kernel to add functiona
|
|
lity or change behavior.
|
|
|
|
\layout Subsection
|
|
|
|
Result Comparison
|
|
\layout Standard
|
|
|
|
A lot of testing will be done in the life of the Linux Test Project.
|
|
Keeping track of the results from all the testing will require some infrastruct
|
|
ure.
|
|
It would be nice to take that output from a test machine, feed it to a
|
|
program and receive a list of items that broke since the last run on that
|
|
machine, or were fixed, or work on another test machine but not on this
|
|
one.
|
|
|
|
\layout Section
|
|
|
|
Contact information and updates
|
|
\layout Literal
|
|
|
|
URL: http://ltp.sourceforge.net/
|
|
\layout Literal
|
|
|
|
mailing list: ltp@lists.linux.it
|
|
\layout Literal
|
|
|
|
list archive: http://lists.linux.it/pipermail/ltp/
|
|
\layout Standard
|
|
|
|
Questions and comments should be sent to the LTP mailing
|
|
list at ltp@lists.linux.it. To subscribe, please go to
|
|
http://lists.linux.it/pipermail/ltp/.
|
|
|
|
\layout Standard
|
|
|
|
The source is also available via CVS.
|
|
See the web site for a web interface and check out instructions.
|
|
|
|
\layout Section
|
|
|
|
Glossary
|
|
\layout Description
|
|
|
|
Test IEEE/ANSI
|
|
\begin_float footnote
|
|
\layout Standard
|
|
|
|
Kit, Edward, Software Testing in the Real World: Improving the Process.
|
|
P.
|
|
82.
|
|
ACM Press, 1995.
|
|
\end_float
|
|
:
|
|
\shape italic
|
|
|
|
\newline
|
|
|
|
\shape default
|
|
|
|
\shape italic
|
|
(i)
|
|
\shape default
|
|
An activity in which a system or component is executed under specified
|
|
conditions, the results are observed or record, and an evaluation is made
|
|
of some aspect of the system or component.
|
|
|
|
\shape italic
|
|
|
|
\newline
|
|
|
|
\shape default
|
|
|
|
\shape italic
|
|
(ii)
|
|
\shape default
|
|
A set of one or more test cases.
|
|
|
|
\layout Description
|
|
|
|
Test\SpecialChar ~
|
|
Case A test assertion with a single result that is being verified.
|
|
This allows designations such as PASS or FAIL to be applied to a single
|
|
bit of functionality.
|
|
A single test case may be one of many test cases for testing the complete
|
|
functionality of a system.
|
|
|
|
\newline
|
|
IEEE/ANSI:
|
|
\shape italic
|
|
|
|
\newline
|
|
(i)
|
|
\shape default
|
|
A set of test inputs, execution conditions, and expected results developed
|
|
for a particular objective.
|
|
|
|
\shape italic
|
|
|
|
\newline
|
|
(ii)
|
|
\shape default
|
|
The smallest entity that is always executed as a unit, from beginning to
|
|
end.
|
|
|
|
\layout Description
|
|
|
|
Test\SpecialChar ~
|
|
Driver A program that handles the execution of test programs.
|
|
It is responsible for starting the test programs, capturing their output,
|
|
and recording their results.
|
|
Ltp-pan is an example of a test driver.
|
|
\layout Description
|
|
|
|
Test\SpecialChar ~
|
|
Framework A mechanism for organizing a group of tests.
|
|
Frameworks may have complex or very simple API's, drivers and result logging
|
|
mechanisms.
|
|
Examples of frameworks are TETware and DejaGnu.
|
|
|
|
\layout Description
|
|
|
|
Test\SpecialChar ~
|
|
Harness A Test harness is the mechanism that connects a test program
|
|
to a test framework.
|
|
It may be a specification of exit codes, or a set of libraries for formatting
|
|
messages and determining exit codes.
|
|
In TETware, the tet_result() API is the test harness.
|
|
|
|
\layout Description
|
|
|
|
Test\SpecialChar ~
|
|
Program A single invokable program.
|
|
A test program can contain one or more test cases.
|
|
The test harness's API allows for reporting/analysis of the individual
|
|
test cases.
|
|
|
|
\layout Description
|
|
|
|
Test\SpecialChar ~
|
|
Suite A collection of tests programs, assertions, cases grouped together
|
|
under a framework.
|
|
|
|
\layout Description
|
|
|
|
Test\SpecialChar ~
|
|
Tag An identifier that corresponds to a command line which runs a test.
|
|
The tag is a single word that matches a test program with a set of command
|
|
line arguments.
|
|
|
|
\the_end
|