479 lines
13 KiB
HTML
479 lines
13 KiB
HTML
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
|
|
|
|
<!--Converted with LaTeX2HTML 99.1 release (March 30, 1999)
|
|
original version by: Nikos Drakos, CBLU, University of Leeds
|
|
* revised and updated by: Marcus Hennecke, Ross Moore, Herb Swan
|
|
* with significant contributions from:
|
|
Jens Lippmann, Marek Rouchal, Martin Wilck and others -->
|
|
<HTML>
|
|
<HEAD>
|
|
<TITLE>Proposed Test Suite Design</TITLE>
|
|
<META NAME="description" CONTENT="Proposed Test Suite Design">
|
|
<META NAME="keywords" CONTENT="test_suite_spec">
|
|
<META NAME="resource-type" CONTENT="document">
|
|
<META NAME="distribution" CONTENT="global">
|
|
|
|
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
|
|
<META NAME="Generator" CONTENT="LaTeX2HTML v99.1 release">
|
|
<META HTTP-EQUIV="Content-Style-Type" CONTENT="text/css">
|
|
|
|
<LINK REL="STYLESHEET" HREF="test_suite_spec.css">
|
|
|
|
</HEAD>
|
|
|
|
<BODY >
|
|
<!--Navigation Panel-->
|
|
<IMG WIDTH="81" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="next_group" SRC="next_group_motif_gr.gif">
|
|
<IMG WIDTH="26" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="up" SRC="up_motif_gr.gif">
|
|
<IMG WIDTH="63" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="previous" SRC="previous_motif_gr.gif">
|
|
<BR>
|
|
<BR>
|
|
<BR>
|
|
<!--End of Navigation Panel-->
|
|
|
|
<P>
|
|
|
|
<P>
|
|
|
|
<P>
|
|
|
|
<P>
|
|
|
|
<H1 ALIGN="CENTER">Proposed Test Suite Design</H1>
|
|
<P ALIGN="CENTER"><STRONG>Michael Hope (michaelh@juju.net.nz)</STRONG></P>
|
|
<P ALIGN="CENTER"><STRONG>13 July 2001</STRONG></P>
|
|
|
|
<H3>Abstract:</H3>
|
|
<DIV>
|
|
This article describes the goals, requirements, and suggested specification
|
|
for a test suite for the output of the Small Device C Compiler (sdcc).
|
|
Also included is a short list of existing works.
|
|
</DIV>
|
|
<P>
|
|
|
|
<P>
|
|
|
|
<H1><A NAME="SECTION00010000000000000000">
|
|
Goals</A>
|
|
</H1>
|
|
|
|
<P>
|
|
The main goals of a test suite for sdcc are
|
|
|
|
<P>
|
|
|
|
<OL>
|
|
<LI>To allow developers to run regression tests to check that core changes
|
|
do not break any of the many ports. </LI>
|
|
<LI>To verify the core. </LI>
|
|
<LI>To allow developers to verify individual ports. </LI>
|
|
<LI>To allow developers to test port changes. </LI>
|
|
</OL>
|
|
This design only covers the generated code. It does not cover a test/unit
|
|
test framework for the sdcc application itself, which may be useful.
|
|
|
|
<P>
|
|
One side effect of (1) is that it requires that the individual ports
|
|
pass the tests originally. This may be too hard. See the section on
|
|
Exceptions below.
|
|
|
|
<P>
|
|
|
|
<H1><A NAME="SECTION00020000000000000000">
|
|
Requirements</A>
|
|
</H1>
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00021000000000000000">
|
|
Coverage</A>
|
|
</H2>
|
|
|
|
<P>
|
|
The suite is intended to cover language features only. Hardware specific
|
|
libraries are explicitly not covered.
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00022000000000000000">
|
|
Permutations</A>
|
|
</H2>
|
|
|
|
<P>
|
|
The ports often generate different code for handling different types
|
|
(Byte, Word, DWord, and the signed forms). Meta information could
|
|
be used to permute the different test cases across the different types.
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00023000000000000000">
|
|
Exceptions</A>
|
|
</H2>
|
|
|
|
<P>
|
|
The different ports are all at different levels of development. Test
|
|
cases must be able to be disabled on a per port basis. Permutations
|
|
also must be able to be disabled on a port level for unsupported cases.
|
|
Disabling, as opposed to enabling, on a per port basis seems more
|
|
maintainable.
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00024000000000000000">
|
|
Running</A>
|
|
</H2>
|
|
|
|
<P>
|
|
The tests must be able to run unaided. The test suite must run on
|
|
all platforms that sdcc runs on. A good minimum may be a subset of
|
|
Unix command set and common tools, provided by default on a Unix host
|
|
and provided through cygwin on a Windows host.
|
|
|
|
<P>
|
|
The tests suits should be able to be sub-divided, so that the failing
|
|
or interesting tests may be run separately.
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00025000000000000000">
|
|
Artifcats</A>
|
|
</H2>
|
|
|
|
<P>
|
|
The test code within the test cases should not generate artifacts.
|
|
An artifact occurs when the test code itself interferes with the test
|
|
and generates an erroneous result.
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00026000000000000000">
|
|
Emulators</A>
|
|
</H2>
|
|
|
|
<P>
|
|
sdcc is a cross compiling compiler. As such, an emulator is needed
|
|
for each port to run the tests.
|
|
|
|
<P>
|
|
|
|
<H1><A NAME="SECTION00030000000000000000">
|
|
Existing works</A>
|
|
</H1>
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00031000000000000000">
|
|
DejaGnu</A>
|
|
</H2>
|
|
|
|
<P>
|
|
DejaGnu is a toolkit written in Expect designed to test an interactive
|
|
program. It provides a way of specifying an interface to the program,
|
|
and given that interface a way of stimulating the program and interpreting
|
|
the results. It was originally written by Cygnus Solutions for running
|
|
against development boards. I believe the gcc test suite is written
|
|
against DejaGnu, perhaps partly to test the Cygnus ports of gcc on
|
|
target systems.
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00032000000000000000">
|
|
gcc test suite</A>
|
|
</H2>
|
|
|
|
<P>
|
|
I don't know much about the gcc test suite. It was recently removed
|
|
from the gcc distribution due to issues with copyright ownership.
|
|
The code I saw from older distributions seemed more concerned with
|
|
esoteric features of the language.
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00033000000000000000">
|
|
xUnit</A>
|
|
</H2>
|
|
|
|
<P>
|
|
The xUnit family, in particular JUnit, is a library of in test assertions,
|
|
test wrappers, and test suite wrappers designed mainly for unit testing.
|
|
PENDING: More.
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00034000000000000000">
|
|
CoreLinux++ Assertion framework</A>
|
|
</H2>
|
|
|
|
<P>
|
|
While not a test suite system, the assertion framework is an interesting
|
|
model for the types of assertions that could be used. They include
|
|
pre-condition, post-condition, invariants, conditional assertions,
|
|
unconditional assertions, and methods for checking conditions.
|
|
|
|
<P>
|
|
|
|
<H1><A NAME="SECTION00040000000000000000">
|
|
Specification</A>
|
|
</H1>
|
|
|
|
<P>
|
|
This specification borrows from the JUnit style of unit testing and
|
|
the CoreLinux++ style of assertions. The emphasis is on maintainability
|
|
and ease of writing the test cases.
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00041000000000000000">
|
|
Terms</A>
|
|
</H2>
|
|
|
|
<P>
|
|
PENDING: Align these terms with the rest of the world.
|
|
|
|
<P>
|
|
|
|
<UL>
|
|
<LI>An <I>assertion</I> is a statement of how things should be. PENDING:
|
|
Better description, an example. </LI>
|
|
<LI>A <I>test point</I> is the smallest unit of a test suite, and consists
|
|
of a single assertion that passes if the test passes. </LI>
|
|
<LI>A <I>test case</I> is a set of test points that test a certain feature. </LI>
|
|
<LI>A <I>test suite</I> is a set of test cases that test a certain set
|
|
of features. </LI>
|
|
</UL>
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00042000000000000000">
|
|
Test cases</A>
|
|
</H2>
|
|
|
|
<P>
|
|
Test cases shall be contained in their own C file, along with the
|
|
meta data on the test. Test cases shall be contained within functions
|
|
whose names start with 'test' and which are descriptive of the test
|
|
case. Any function that starts with 'test' will be automatically run
|
|
in the test suite.
|
|
|
|
<P>
|
|
To make the automatic code generation easier, the C code shall have
|
|
this format
|
|
|
|
<P>
|
|
|
|
<UL>
|
|
<LI>Test functions shall start with 'test' to allow automatic detection. </LI>
|
|
<LI>Test functions shall follow the K&R intention style for ease of detection.
|
|
i.e. the function name shall start in the left column on a new line
|
|
below the return specification. </LI>
|
|
</UL>
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00043000000000000000">
|
|
Assertions</A>
|
|
</H2>
|
|
|
|
<P>
|
|
All assertions shall log the line number, function name, and test
|
|
case file when they fail. Most assertions can have a more descriptive
|
|
message attached to them. Assertions will be implemented through macros
|
|
to get at the line information. This may cause trouble with artifacts.
|
|
|
|
<P>
|
|
The following definitions use C++ style default arguments where optional
|
|
messages may be inserted. All assertions use double opening and closing
|
|
brackets in the macros to allow them to be compiled out without any
|
|
side effects. While this is not required for a test suite, they are
|
|
there in case any of this code is incorporated into the main product.
|
|
|
|
<P>
|
|
Borrowing from JUnit, the assertions shall include
|
|
|
|
<P>
|
|
|
|
<UL>
|
|
<LI>FAIL((String msg = ``Failed'')). Used when execution should not
|
|
get here. </LI>
|
|
<LI>ASSERT((Boolean cond, String msg = ``Assertion failed''). Fails
|
|
if cond is false. Parent to REQUIRE and ENSURE. </LI>
|
|
</UL>
|
|
JUnit also includes may sub-cases of ASSERT, such as assertNotNull,
|
|
assertEquals, and assertSame.
|
|
|
|
<P>
|
|
CoreLinux++ includes the extra assertions
|
|
|
|
<P>
|
|
|
|
<UL>
|
|
<LI>REQUIRE((Boolean cond, String msg = ``Precondition failed'').
|
|
Checks preconditions. </LI>
|
|
<LI>ENSURE((Boolean cond, String msg = ``Postcondition failed'').
|
|
Checks post conditions. </LI>
|
|
<LI>CHECK((Boolean cond, String msg = ``Check failed'')). Used to
|
|
call a function and to check that the return value is as expected.
|
|
i.e. CHECK((fread(in, buf, 10) != -1)). Very similar to ASSERT, but
|
|
the function still gets called in a release build. </LI>
|
|
<LI>FORALL and EXISTS. Used to check conditions within part of the code.
|
|
For example, can be used to check that a list is still sorted inside
|
|
each loop of a sort routine. </LI>
|
|
</UL>
|
|
All of FAIL, ASSERT, REQUIRE, ENSURE, and CHECK shall be available.
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00044000000000000000">
|
|
Meta data</A>
|
|
</H2>
|
|
|
|
<P>
|
|
PENDING: It's not really meta data.
|
|
|
|
<P>
|
|
Meta data includes permutation information, exception information,
|
|
and permutation exceptions.
|
|
|
|
<P>
|
|
Meta data shall be global to the file. Meta data names consist of
|
|
the lower case alphanumerics. Test case specific meta data (fields)
|
|
shall be stored in a comment block at the start of the file. This
|
|
is only due to style.
|
|
|
|
<P>
|
|
A field definition shall consist of
|
|
|
|
<P>
|
|
|
|
<UL>
|
|
<LI>The field name </LI>
|
|
<LI>A colon. </LI>
|
|
<LI>A comma separated list of values. </LI>
|
|
</UL>
|
|
The values shall be stripped of leading and trailing white space.
|
|
|
|
<P>
|
|
Permutation exceptions are by port only. Exceptions to a field are
|
|
specified by a modified field definition. An exception definition
|
|
consists of
|
|
|
|
<P>
|
|
|
|
<UL>
|
|
<LI>The field name. </LI>
|
|
<LI>An opening square bracket. </LI>
|
|
<LI>A comma separated list of ports the exception applies for. </LI>
|
|
<LI>A closing square bracket. </LI>
|
|
<LI>A colon. </LI>
|
|
<LI>The values to use for this field for these ports. </LI>
|
|
</UL>
|
|
An instance of the test case shall be generated for each permutation
|
|
of the test case specific meta data fields.
|
|
|
|
<P>
|
|
The runtime meta fields are
|
|
|
|
<P>
|
|
|
|
<UL>
|
|
<LI>port - The port this test is running on. </LI>
|
|
<LI>testcase - The name of this test case. </LI>
|
|
<LI>function - The name of the current function. </LI>
|
|
</UL>
|
|
Most of the runtime fields are not very usable. They are there for
|
|
completeness.
|
|
|
|
<P>
|
|
Meta fields may be accessed inside the test case by enclosing them
|
|
in curly brackets. The curly brackets will be interpreted anywhere
|
|
inside the test case, including inside quoted strings. Field names
|
|
that are not recognised will be passed through including the brackets.
|
|
Note that it is therefore impossible to use some strings within the
|
|
test case.
|
|
|
|
<P>
|
|
Test case function names should include the permuted fields in the
|
|
name to reduce name collisions.
|
|
|
|
<P>
|
|
|
|
<H2><A NAME="SECTION00045000000000000000">
|
|
An example</A>
|
|
</H2>
|
|
|
|
<P>
|
|
I don't know how to do pre-formatted text in L<SUP><SMALL>A</SMALL></SUP>T<SMALL>E</SMALL>X . Sigh.
|
|
|
|
<P>
|
|
The following code generates a simple increment test for all combinations
|
|
of the storage classes and all combinations of the data sizes. This
|
|
is a bad example as the optimiser will often remove most of this code.
|
|
|
|
<P>
|
|
<TT>/** Test for increment. </TT>
|
|
|
|
<P>
|
|
<TT>type: char, int, long</TT>
|
|
|
|
<P>
|
|
<TT>Z80 port does not fully support longs (4 byte)</TT>
|
|
|
|
<P>
|
|
<TT>type[z80]: char, int</TT>
|
|
|
|
<P>
|
|
<TT>class: ``'', register, static */</TT>
|
|
|
|
<P>
|
|
<TT>static void</TT>
|
|
|
|
<P>
|
|
<TT>testInc{class}{types}(void)</TT>
|
|
|
|
<P>
|
|
<TT>{</TT>
|
|
|
|
<P>
|
|
<TT>{class} {type} i = 0; </TT>
|
|
|
|
<P>
|
|
<TT>i = i + 1;</TT>
|
|
|
|
<P>
|
|
<TT>ASSERT((i == 1));</TT>
|
|
|
|
<P>
|
|
<TT>}</TT>
|
|
|
|
<H1><A NAME="SECTION00050000000000000000">
|
|
About this document ...</A>
|
|
</H1>
|
|
<STRONG>Proposed Test Suite Design</STRONG><P>
|
|
This document was generated using the
|
|
<A HREF="http://www-dsed.llnl.gov/files/programs/unix/latex2html/manual/"><STRONG>LaTeX</STRONG>2<tt>HTML</tt></A> translator Version 99.1 release (March 30, 1999)
|
|
<P>
|
|
Copyright © 1993, 1994, 1995, 1996,
|
|
<A HREF="http://cbl.leeds.ac.uk/nikos/personal.html">Nikos Drakos</A>,
|
|
Computer Based Learning Unit, University of Leeds.
|
|
<BR>
|
|
Copyright © 1997, 1998, 1999,
|
|
<A HREF="http://www.maths.mq.edu.au/~ross/">Ross Moore</A>,
|
|
Mathematics Department, Macquarie University, Sydney.
|
|
<P>
|
|
The command line arguments were: <BR>
|
|
<STRONG>latex2html</STRONG> <TT>-split 0 -dir test_suite_spec.html test_suite_spec</TT>
|
|
<P>
|
|
The translation was initiated by Johan Knol on 2001-07-13<HR>
|
|
<!--Navigation Panel-->
|
|
<IMG WIDTH="81" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="next_group" SRC="next_group_motif_gr.gif">
|
|
<IMG WIDTH="26" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="up" SRC="up_motif_gr.gif">
|
|
<IMG WIDTH="63" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="previous" SRC="previous_motif_gr.gif">
|
|
<BR>
|
|
<!--End of Navigation Panel-->
|
|
<ADDRESS>
|
|
<I>Johan Knol</I>
|
|
<BR><I>2001-07-13</I>
|
|
</ADDRESS>
|
|
</BODY>
|
|
</HTML>
|