Fit Specification: Fixtures

 

Fit documents are owned by the customers and domain experts on your team.  The documents contain examples of how the program should work and Fit checks the examples against the actual program.  The examples are placed in tables with a special first row, but other than that, there are no limitations on how the examples are structured.  Customers should write tables in the way that comes most naturally to them.

 

Since there aren’t any limits to the way tables are structured, Fit can’t interpret the tables automatically.  It requires a fixture to interpret the table.  A fixture is a piece of code, written by the team’s programmers, that understands the structure of each type of table and how to check it against the actual program.

 

Different implementations of Fit may use different mechanisms for fixtures.  For example, the Java version of Fit uses classes to implement fixtures, but other versions of Fit may use a different mechanism.

 

Regardless of how fixtures are implemented, they have some common elements, described in more detail below.  There are also several predefined types of fixtures which programmers may specialize, listed in the table below.  But before you look at the details of each type of fixture, read the rest of this document to learn the functionality common to all fixtures.

 

fat.ReferenceFixture

 

 

Description

Location

Result()

Primitive Fixture

fixtures/primitive.html

 

Column Fixture

fixtures/column.html

 

Row Fixture

fixtures/row.html

 

Action Fixture

fixtures/action.html

 

Summary Fixture

fixtures/summary.html

 

 

Fixture Names

 

The special first row of a table is the name of the fixture used to interpret that table.  There’s no special logic applied; if the first cell in the first row is not the correct fixture name, Fit won’t try to guess where it is.

 

fat.FixtureNameFixture

 

Table

FixtureName()

[exampleFixture]

exampleFixture

[exampleFixture]

[1] [2]

[3] [4]

exampleFixture

[] [exampleFixture]

blank

[exampleFixture] [foo]

exampleFixture

[]

[exampleFixture]

blank

 

The fixture name is a direct reference to the class or other piece of code that will interpret the fixture.  It’s fully qualified, containing both the namespace and name of the fixture.  Fit uses reflection (or its equivalent) to instantiate the fixture.  The fixture name is case sensitive and must be fully qualified.

 

fat.FixtureNameFixture

 

Table

ValidFixture()

[fat.FixtureNameFixture.ExampleFixture]

true

[fat.FixtureNameFixture.exampleFixture]

false

[Fat.FixtureNameFixture.ExampleFixture]

false

[ExampleFixture]

false

[FixtureNameFixture.ExampleFixture]

false

[fat.ExampleFixture]

false

[fat.FixtureNameFixture.Example]

false

 

When the fixture isn’t valid, Fit annotates the cell with an explanatory error message.  (See parsing and output for more information about annotations.) What makes something a fixture or not depends on which implementation of Fit you’re using.  In Java, for example, only classes that extend the “Fixture” class are fixtures.  To continue the example, languages without strict typing might use the presence of specific methods to determine whether something is a fixture or not.

 

fat.FixtureNameFixture

 

 

Table

ValidFixture()

Error()

[fat.FixtureNameFixture.ExampleFixture]

true

blank

[ExampleFixture]

false

The fixture "ExampleFixture" was not found.

[fat.FixtureNameFixture.NotAFixture]

false

"fat.FixtureNameFixture.NotAFixture” was found, but it’s not a fixture.

 

 

 

 

Annotation

 

When a fixture interprets a table, it can modify the table.  This is called “annotation.”  For standard fixtures, each cell in a table is annotated with one of the following annotations:

 

·        none: Fit used the cell but didn’t check its value

·        right: Fit checked the cell against the software and the software gave the correct answer

·        wrong: Fit checked the cell against the software and the software gave the wrong answer   (Changed formatting from official version of Fit)

·        error: Fit checked the cell against the software and the software generated a run-time error (e.g., an exception)   (Official version expects error to be an exception, not text as I imply in the table)

·        ignore: Fit ignored the cell entirely

·        info: Useful information

 

fat.AnnotationFixture

 

 

 

Type

OriginalCell

Text

Output()

none

Text

blank

<td>Text</td>

right

Text

blank

<td bgcolor="#cfffcf">Text</td>

wrong

Text

failure message

<td bgcolor="#ffcfcf"><table><tr><td><font size=-1 color=#400000><i>|</i></font>Text<font size=-1 color=#400000><i>|</i></font></td><td><font size=-1 color=#400000><i>expected</i></font></td></tr><tr><td colspan='2'><hr /></td></tr><tr><td><font size=-1 color=#400000><i>|</i></font>failure message<font size=-1 color=#400000><i>|</i></font></td><td><font size=-1 color=#400000><i>actual</i></font></td></tr></table></td>

error

Text

error message

<td bgcolor="#ffffcf">Text <hr><pre><font size=-2>error message</font></pre></td>

ignore

Text

blank

<td bgcolor="#efefef”>Text</td>

info

Text

value

<td>Text <font color="#808080">value</font></td>

 

Text

 

 

 

When annotating “wrong” and “error” cells, Fit strips mark-up from the contents of the cell so troubleshooting is easier.  It doesn’t do so for other annotations.

 

fat.AnnotationFixture

 

 

 

 

Type

OriginalCell

OriginalCell

Text

Output()

none

<ul><li>Text</li></ul>

<ul><li>Text</li></ul>

blank

<td><ul><li>Text</li></ul></td>

right

<ul><li>Text</li></ul>

<ul><li>Text</li></ul>

blank

<td bgcolor="#cfffcf"><ul><li>Text</li></ul></td>

wrong

<ul><li>Text</li></ul>

<ul><li>Text</li></ul>

failure message

<td bgcolor="#ffcfcf"><table><tr><td><font size=-1 color=#400000><i>|</i></font>Text<font size=-1 color=#400000><i>|</i></font></td><td><font size=-1 color=#400000><i>expected</i></font></td></tr><tr><td colspan='2'><hr /></td></tr><tr><td><font size=-1 color=#400000><i>|</i></font>failure message<font size=-1 color=#400000><i>|</i></font></td><td><font size=-1 color=#400000><i>actual</i></font></td></tr></table></td>

error

<ul><li>Text</li></ul>

<ul><li>Text</li></ul>

error message

<td bgcolor="#ffffcf">Text <hr><pre><font size=-2>error message</font></pre></td>

 

error

<ul><li>Line breaks<br />are preserved</li></ul>

<ul><li>Line breaks<br />are preserved</li></ul>

error message

<td bgcolor="#ffffcf">Line breaks<br />are preserved <hr><pre><font size=-2>error message</font></pre></td>

 

ignore

<ul><li>Text</li></ul>

<ul><li>Text</li></ul>

blank

<td bgcolor="#efefef”><ul><li>Text</li></ul></td>

info

<ul><li>Text</li></ul>

<ul><li>Text</li></ul>

value

<td><ul><li>Text</li></ul> <font color="#808080">value</font></td>

 

 

In addition to the specific annotations described above, fixtures may also modify tables directly.  This gives them the ability to introduce new kinds of annotations:

 

fat.AnnotationFixture

 

 

Type

Text

Output()

none

 

<td>Text</td>

body

New body

<td>New body</td>

body

Anything <i>you</i> want!

<td>Anything <i>you</i> want!</td>

tag

<new tag>

<new tag>Text</new tag>

tag

<td align="center">

<td align="center">Text</td>

addToBody

<br />New body

<td>Text<br />New body</td>

addToTag

bgcolor="red"

<td bgcolor="red">Text</td>

 

Although the above examples use specific names for the different types of annotations, the code used by the fixtures may be different.

 

 

Return to the beginning.

 

 

Errata:

 

Known errors and omissions (fix me!):

 

 

 

Possible improvements:

 

 

 

 

 

fit.Summary

 

 

The parsed tables can be modified in memory and a revised document written.

·         cell background color

·         cell contents

·         additional rows

·         additional columns


Check

Expected values specified in cells are tested for equality with actual values extracted from the program under test.

fat.Equals

type

x

y

=

boolean

true

TRUE

true

integer

00001

1

true

real

1000

1e3

true

string

abc

ABC

false

string

a b c

a b c

true

A sequence of values can be entered and checked as a single (composite) value.

fat.Equals

 

 

 

type

x

y

=

integers

1, 3, 5

01,03,05

true

integers

1, 3, 5

01,05,03

false

booleans

true, true, false

true, false

false

strings

a , b , c

a,b,c

true

Domain values can be constructed from cell contents. Equality comparisons will be subject to appropriate domain rules (i.e. delegated to the domain objects.)

fat.Equals

type

x

y

=

date

Jan 1, 1995

January 1, 1995

true

money

$10000

$10,000.00

true

Floating point numbers are checked to the precision that is normal in the host language. It is possible for a domain object to infer precision from the string representation of expected values. For example, ?ScientificDouble checks equality to the precision implied by the number of significant digits in a value.

·         non-zeros -- 123.45 is 5 digits

·         zeros between non-zeros -- 100003 is 6 digits

·         zeros in the fraction part -- 12.20 is 4 digits

·         but not leading zeros -- 0001000 is 4 digits

·         and not exponent digits -- 6.02e23 is 3 digits

·         left argument (receiver) controles precision

fat.Equals

type

x

y

=

real

123.45

123.449

false

scientific

123.45

123.449

true

scientific

100003

100003.1

true

scientific

100003.1

100003

false

scientific

12.20

12.210

false

scientific

12.21

12.210

true

Improperly specified values may throw exceptions which are reported in the cell that contains the invalid number. Improper value detection is only as good as would be expected in the host language. (We are assuming that incorrectly converted values will lead to detected errors elsewhere.)

·         xyz is not an integer

·         10000000000000000000000000 may or may not be an integer

fat.Table

fat.Equals

 

type

x

integer

200

integer

xyz

 

fat.Color

white

white

white

white

white

white

white

yellow

Some character strings have special meaning outside of type conversion.

·         blank -- omit check, report value (reported in gray letters)

·         error -- expect an error

fat.Table

fat.Divide

 

 

x

y

divide()

100

2

50

100

2

 

100

0

error

100

0

 

 

fat.Color

white

white

white

white

white

white

white

white

green

white

white

gray/white

white

white

green

white

white

gray/white

The framework looks for parse support in different places.

·         fixture

·         domain object

·         type adapter

The framework will allow fixtures to implement custom parsing.

·         yea = true

·         nay = false


Fixture

The first cell of a table specifies how that table will be interpreted. Normally this is the name of an object, a Fixture, that is specifically written for this purpose.

·         case Foo != foo

·         package alpha.Foo != beta.Foo

By default cells are processed in order, by table, by row within tables, and by cell within rows. Distinct routines handle sequencing (plural) and interpretation (singular).

·         doTables -- sequence tables within a document

·         doTable -- interpret a table

·         doRows -- sequence rows within a table

·         doRow -- interpret a row

·         doCells -- sequence cells within a row

·         doCell -- interpret a row

The default interpretation of a cell is to mark it as ignored.

Cells contain givens or expected results. The fixture is responsible for distinguishing givens from expected results, checking expected results against actuals, and indicating the result with the background annotation as follows.

·         white -- no check wanted (blank input)

·         green -- right: expected equals actual

·         red -- wrong: expected not equal actual

·         yellow -- exception: trouble computing actual or comparing result

·         gray -- ignore: desired check not possible

Fixtures cooperate to count the number of each result.

·         counts accumulate between tables

·         counts are summarized as "# right, # wrong, # exceptions, # ignores".

·         runner fixtures (e.g. AllFiles) can control accumulation of counts

Fixtures cooperate to accumulate state that can be summarized at any point in the processing of a document.

·         state variables

·         state lifetime

·         state summary


PrimitiveFixture

The primitive fixture does not depend on TypeAdapter. Type specific functions handle all checking.

·         checkBoolean

·         checkInteger

·         checkFloat

·         checkString


ColumnFixture

A column fixture distinguishes givens from expected values by the form of the column head.

·         name is setter or field

·         name() is getter or field

·         camel case -- present value ( ) = presentValue()

·         deprecate execute()


RowFixture

A row fixture expects all rows to be expected values. The fixture is assumed to have access to actual rows which it pairs with expected rows.

·         name is field or getter

·         name() is getter

·         camel case

Rows are matched by comparing just enough values from the left most columns to get a unique match.

·         some given, none actual -- missing row

·         none given, some actual -- surplus rows

·         one given, one actual -- check remaining columns

·         many given, many actual -- match on more columns

Missing and surplus rows are marked as such in their first column

·         each counts as one wrong

·         surplus rows added to table in gray


ActionFixture

An action fixture distinguishes given from expected values based on keywords in the first column.

·         enter -- put given in setter or field

·         check -- check value with getter or field

·         press -- call action (no check other than for execptions)

·         start -- new actor

An actor represents some interface state such as the currently active window. The actor gives meaning to many of the words in the second column (field to be entered and buttons to be pressed).

·         actor persists between tables

·         initial actor is action fixture (subclass) itself

·         start with no argument specifies initial actor

Start could have various meanings depending on the nature of the interface being modeled.

·         create a new actor

·         return to an existing actor

Other actions, like press, can change the current actor.

·         replace the current actor with a new actor

·         replace the current actor with another preexisting actor

·         restore the previous actor

Subclasses of action fixture can add new action words.

·         camel case


Runner

A runner creates top-level Fixtures and provides them with parsed documents to be annotated.

·         top-level fixture is instance of Fixture

A runner retrieves and disposes of documents as appropriate to its environment. The FileRunner reads and writes html files.

·         input-file only -- default output

·         input-file output-file -- specific output

·         input-directory only -- default output in directory

·         input-directory output-directory -- default outputs in specific directory

·         index.html result summaries

·         - is standard in/out

A runner signals the presence of errors to its environment so that subsequent processing can be triggered.

·         exit status is sum of wrong and exceptions up to 255

·         main is factored to simplify ?RunnerFixtures

A runner establishes run specific state that may influence loading and/or configuration of the program under test.

·         ?AssemblyPath ???