544 lines
11 KiB
Plaintext
544 lines
11 KiB
Plaintext
#LyX 1.1 created this file. For more info see http://www.lyx.org/
|
|
\lyxformat 218
|
|
\textclass article
|
|
\begin_preamble
|
|
\usepackage{url}
|
|
\end_preamble
|
|
\language english
|
|
\inputencoding auto
|
|
\fontscheme times
|
|
\graphics default
|
|
\paperfontsize default
|
|
\spacing single
|
|
\papersize Default
|
|
\paperpackage a4
|
|
\use_geometry 0
|
|
\use_amsmath 0
|
|
\paperorientation portrait
|
|
\secnumdepth 3
|
|
\tocdepth 3
|
|
\paragraph_separation indent
|
|
\defskip medskip
|
|
\quotes_language english
|
|
\quotes_times 2
|
|
\papercolumns 1
|
|
\papersides 1
|
|
\paperpagestyle default
|
|
|
|
\layout Title
|
|
|
|
Proposed Test Suite Design
|
|
\layout Author
|
|
|
|
Michael Hope (michaelh@juju.net.nz)
|
|
\layout Date
|
|
|
|
|
|
\latex latex
|
|
|
|
\backslash
|
|
today{}
|
|
\layout Abstract
|
|
|
|
This article describes the goals, requirements, and suggested specification
|
|
for a test suite for the output of the Small Device C Compiler (sdcc).
|
|
Also included is a short list of existing works.
|
|
|
|
\layout Section
|
|
|
|
Goals
|
|
\layout Standard
|
|
|
|
The main goals of a test suite for sdcc are
|
|
\layout Enumerate
|
|
|
|
To allow developers to run regression tests to check that core changes do
|
|
not break any of the many ports.
|
|
|
|
\layout Enumerate
|
|
|
|
To verify the core.
|
|
|
|
\layout Enumerate
|
|
|
|
To allow developers to verify individual ports.
|
|
|
|
\layout Enumerate
|
|
|
|
To allow developers to test port changes.
|
|
|
|
\layout Standard
|
|
|
|
This design only covers the generated code.
|
|
It does not cover a test/unit test framework for the sdcc application itself,
|
|
which may be useful.
|
|
\layout Standard
|
|
|
|
One side effect of (1) is that it requires that the individual ports pass
|
|
the tests originally.
|
|
This may be too hard.
|
|
See the section on Exceptions below.
|
|
\layout Section
|
|
|
|
Requirements
|
|
\layout Subsection
|
|
|
|
Coverage
|
|
\layout Standard
|
|
|
|
The suite is intended to cover language features only.
|
|
Hardware specific libraries are explicitly not covered.
|
|
\layout Subsection
|
|
|
|
Permutations
|
|
\layout Standard
|
|
|
|
The ports often generate different code for handling different types (Byte,
|
|
Word, DWord, and the signed forms).
|
|
Meta information could be used to permute the different test cases across
|
|
the different types.
|
|
\layout Subsection
|
|
|
|
Exceptions
|
|
\layout Standard
|
|
|
|
The different ports are all at different levels of development.
|
|
Test cases must be able to be disabled on a per port basis.
|
|
Permutations also must be able to be disabled on a port level for unsupported
|
|
cases.
|
|
Disabling, as opposed to enabling, on a per port basis seems more maintainable.
|
|
\layout Subsection
|
|
|
|
Running
|
|
\layout Standard
|
|
|
|
The tests must be able to run unaided.
|
|
The test suite must run on all platforms that sdcc runs on.
|
|
A good minimum may be a subset of Unix command set and common tools, provided
|
|
by default on a Unix host and provided through cygwin on a Windows host.
|
|
\layout Standard
|
|
|
|
The tests suits should be able to be sub-divided, so that the failing or
|
|
interesting tests may be run separately.
|
|
\layout Subsection
|
|
|
|
Artifcats
|
|
\layout Standard
|
|
|
|
The test code within the test cases should not generate artifacts.
|
|
An artifact occurs when the test code itself interferes with the test and
|
|
generates an erroneous result.
|
|
\layout Subsection
|
|
|
|
Emulators
|
|
\layout Standard
|
|
|
|
sdcc is a cross compiling compiler.
|
|
As such, an emulator is needed for each port to run the tests.
|
|
\layout Section
|
|
|
|
Existing works
|
|
\layout Subsection
|
|
|
|
DejaGnu
|
|
\layout Standard
|
|
|
|
DejaGnu is a toolkit written in Expect designed to test an interactive program.
|
|
It provides a way of specifying an interface to the program, and given
|
|
that interface a way of stimulating the program and interpreting the results.
|
|
It was originally written by Cygnus Solutions for running against development
|
|
boards.
|
|
I believe the gcc test suite is written against DejaGnu, perhaps partly
|
|
to test the Cygnus ports of gcc on target systems.
|
|
\layout Subsection
|
|
|
|
gcc test suite
|
|
\layout Standard
|
|
|
|
I don't know much about the gcc test suite.
|
|
It was recently removed from the gcc distribution due to issues with copyright
|
|
ownership.
|
|
The code I saw from older distributions seemed more concerned with esoteric
|
|
features of the language.
|
|
\layout Subsection
|
|
|
|
xUnit
|
|
\layout Standard
|
|
|
|
The xUnit family, in particular JUnit, is a library of in test assertions,
|
|
test wrappers, and test suite wrappers designed mainly for unit testing.
|
|
PENDING: More.
|
|
\layout Subsection
|
|
|
|
CoreLinux++ Assertion framework
|
|
\layout Standard
|
|
|
|
While not a test suite system, the assertion framework is an interesting
|
|
model for the types of assertions that could be used.
|
|
They include pre-condition, post-condition, invariants, conditional assertions,
|
|
unconditional assertions, and methods for checking conditions.
|
|
\layout Section
|
|
|
|
Specification
|
|
\layout Standard
|
|
|
|
This specification borrows from the JUnit style of unit testing and the
|
|
CoreLinux++ style of assertions.
|
|
The emphasis is on maintainability and ease of writing the test cases.
|
|
\layout Subsection
|
|
|
|
Terms
|
|
\layout Standard
|
|
|
|
PENDING: Align these terms with the rest of the world.
|
|
\layout Itemize
|
|
|
|
An
|
|
\emph on
|
|
assertion
|
|
\emph default
|
|
is a statement of how things should be.
|
|
PENDING: Better description, an example.
|
|
|
|
\layout Itemize
|
|
|
|
A
|
|
\emph on
|
|
test point
|
|
\emph default
|
|
is the smallest unit of a test suite, and consists of a single assertion
|
|
that passes if the test passes.
|
|
|
|
\layout Itemize
|
|
|
|
A
|
|
\emph on
|
|
test case
|
|
\emph default
|
|
is a set of test points that test a certain feature.
|
|
|
|
\layout Itemize
|
|
|
|
A
|
|
\emph on
|
|
test suite
|
|
\emph default
|
|
is a set of test cases that test a certain set of features.
|
|
|
|
\layout Subsection
|
|
|
|
Test cases
|
|
\layout Standard
|
|
|
|
Test cases shall be contained in their own C file, along with the meta data
|
|
on the test.
|
|
Test cases shall be contained within functions whose names start with 'test'
|
|
and which are descriptive of the test case.
|
|
Any function that starts with 'test' will be automatically run in the test
|
|
suite.
|
|
\layout Standard
|
|
|
|
To make the automatic code generation easier, the C code shall have this
|
|
format
|
|
\layout Itemize
|
|
|
|
Test functions shall start with 'test' to allow automatic detection.
|
|
|
|
\layout Itemize
|
|
|
|
Test functions shall follow the K&R intention style for ease of detection.
|
|
i.e.
|
|
the function name shall start in the left column on a new line below the
|
|
return specification.
|
|
|
|
\layout Subsection
|
|
|
|
Assertions
|
|
\layout Standard
|
|
|
|
All assertions shall log the line number, function name, and test case file
|
|
when they fail.
|
|
Most assertions can have a more descriptive message attached to them.
|
|
Assertions will be implemented through macros to get at the line information.
|
|
This may cause trouble with artifacts.
|
|
\layout Standard
|
|
|
|
The following definitions use C++ style default arguments where optional
|
|
messages may be inserted.
|
|
All assertions use double opening and closing brackets in the macros to
|
|
allow them to be compiled out without any side effects.
|
|
While this is not required for a test suite, they are there in case any
|
|
of this code is incorporated into the main product.
|
|
\layout Standard
|
|
|
|
Borrowing from JUnit, the assertions shall include
|
|
\layout Itemize
|
|
|
|
FAIL((String msg =
|
|
\begin_inset Quotes eld
|
|
\end_inset
|
|
|
|
Failed
|
|
\begin_inset Quotes erd
|
|
\end_inset
|
|
|
|
)).
|
|
Used when execution should not get here.
|
|
|
|
\layout Itemize
|
|
|
|
ASSERT((Boolean cond, String msg =
|
|
\begin_inset Quotes eld
|
|
\end_inset
|
|
|
|
Assertion failed
|
|
\begin_inset Quotes erd
|
|
\end_inset
|
|
|
|
).
|
|
Fails if cond is false.
|
|
Parent to REQUIRE and ENSURE.
|
|
|
|
\layout Standard
|
|
|
|
JUnit also includes may sub-cases of ASSERT, such as assertNotNull, assertEquals
|
|
, and assertSame.
|
|
\layout Standard
|
|
|
|
CoreLinux++ includes the extra assertions
|
|
\layout Itemize
|
|
|
|
REQUIRE((Boolean cond, String msg =
|
|
\begin_inset Quotes eld
|
|
\end_inset
|
|
|
|
Precondition failed
|
|
\begin_inset Quotes erd
|
|
\end_inset
|
|
|
|
).
|
|
Checks preconditions.
|
|
|
|
\layout Itemize
|
|
|
|
ENSURE((Boolean cond, String msg =
|
|
\begin_inset Quotes eld
|
|
\end_inset
|
|
|
|
Postcondition failed
|
|
\begin_inset Quotes erd
|
|
\end_inset
|
|
|
|
).
|
|
Checks post conditions.
|
|
|
|
\layout Itemize
|
|
|
|
CHECK((Boolean cond, String msg =
|
|
\begin_inset Quotes eld
|
|
\end_inset
|
|
|
|
Check failed
|
|
\begin_inset Quotes erd
|
|
\end_inset
|
|
|
|
)).
|
|
Used to call a function and to check that the return value is as expected.
|
|
i.e.
|
|
CHECK((fread(in, buf, 10) != -1)).
|
|
Very similar to ASSERT, but the function still gets called in a release
|
|
build.
|
|
|
|
\layout Itemize
|
|
|
|
FORALL and EXISTS.
|
|
Used to check conditions within part of the code.
|
|
For example, can be used to check that a list is still sorted inside each
|
|
loop of a sort routine.
|
|
|
|
\layout Standard
|
|
|
|
All of FAIL, ASSERT, REQUIRE, ENSURE, and CHECK shall be available.
|
|
\layout Subsection
|
|
|
|
Meta data
|
|
\layout Standard
|
|
|
|
PENDING: It's not really meta data.
|
|
\layout Standard
|
|
|
|
Meta data includes permutation information, exception information, and permutati
|
|
on exceptions.
|
|
\layout Standard
|
|
|
|
Meta data shall be global to the file.
|
|
Meta data names consist of the lower case alphanumerics.
|
|
Test case specific meta data (fields) shall be stored in a comment block
|
|
at the start of the file.
|
|
This is only due to style.
|
|
\layout Standard
|
|
|
|
A field definition shall consist of
|
|
\layout Itemize
|
|
|
|
The field name
|
|
\layout Itemize
|
|
|
|
A colon.
|
|
|
|
\layout Itemize
|
|
|
|
A comma separated list of values.
|
|
|
|
\layout Standard
|
|
|
|
The values shall be stripped of leading and trailing white space.
|
|
\layout Standard
|
|
|
|
Permutation exceptions are by port only.
|
|
Exceptions to a field are specified by a modified field definition.
|
|
An exception definition consists of
|
|
\layout Itemize
|
|
|
|
The field name.
|
|
|
|
\layout Itemize
|
|
|
|
An opening square bracket.
|
|
|
|
\layout Itemize
|
|
|
|
A comma separated list of ports the exception applies for.
|
|
|
|
\layout Itemize
|
|
|
|
A closing square bracket.
|
|
|
|
\layout Itemize
|
|
|
|
A colon.
|
|
|
|
\layout Itemize
|
|
|
|
The values to use for this field for these ports.
|
|
|
|
\layout Standard
|
|
|
|
An instance of the test case shall be generated for each permutation of
|
|
the test case specific meta data fields.
|
|
\layout Standard
|
|
|
|
The runtime meta fields are
|
|
\layout Itemize
|
|
|
|
port - The port this test is running on.
|
|
|
|
\layout Itemize
|
|
|
|
testcase - The name of this test case.
|
|
|
|
\layout Itemize
|
|
|
|
function - The name of the current function.
|
|
|
|
\layout Standard
|
|
|
|
Most of the runtime fields are not very usable.
|
|
They are there for completeness.
|
|
\layout Standard
|
|
|
|
Meta fields may be accessed inside the test case by enclosing them in curly
|
|
brackets.
|
|
The curly brackets will be interpreted anywhere inside the test case, including
|
|
inside quoted strings.
|
|
Field names that are not recognised will be passed through including the
|
|
brackets.
|
|
Note that it is therefore impossible to use some strings within the test
|
|
case.
|
|
\layout Standard
|
|
|
|
Test case function names should include the permuted fields in the name
|
|
to reduce name collisions.
|
|
\layout Subsection
|
|
|
|
An example
|
|
\layout Standard
|
|
|
|
I don't know how to do pre-formatted text in LaTeX.
|
|
Sigh.
|
|
\layout Standard
|
|
|
|
The following code generates a simple increment test for all combinations
|
|
of the storage classes and all combinations of the data sizes.
|
|
This is a bad example as the optimiser will often remove most of this code.
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
/** Test for increment.
|
|
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
type: char, int, long
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
Z80 port does not fully support longs (4 byte)
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
type[z80]: char, int
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
class:
|
|
\begin_inset Quotes eld
|
|
\end_inset
|
|
|
|
|
|
\begin_inset Quotes erd
|
|
\end_inset
|
|
|
|
, register, static */
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
static void
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
testInc{class}{types}(void)
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
{
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
{class} {type} i = 0;
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
i = i + 1;
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
ASSERT((i == 1));
|
|
\layout Standard
|
|
|
|
|
|
\family typewriter
|
|
}
|
|
\the_end
|