Autotest 2.0 Help File

It's newer and guier.

Overview

Autotest is a system for automating some of the tasks involved in testing software assignments. Specifically Autotest will

control the compilation of test bench code,
control the compilation of assignments,
run a number of tests,
collect the output from tests as well as whether the test was passed or not,
write a report for each student and also provide the student with the collected output from the tests, and
do all this for all assignments, or a selected subset of the assignment.

There is a lot that can go wrong in the above process and Autotest is capable of dealing with

missing assignments,
assignments that won't compile,
assignments that crash,
assignments that loop forever.

There are a number of tasks that you might think Autotest ought to do but doesn't.

It doesn't help you compare the outputs of submitted code with expected outputs. This is done by an external program you must write. Autotest does however come with a C++ program (test.cpp) that helps you do this, when the students submit (C or C++) program components rather than full programs.
It does not compose a summary file (such as a spreadsheet) of marks. This is partly a matter of philosophy, although I'll probably add the feature if enough people ask me to. The philosophical reason is that I think that assigning a mark is the job of a TA or an instructor, not something that programs should do. My marking schemes usually assign 50% for program correctness and 50% for style. The 50% for correctness is assigned by the TA based on the results of Autotest and also on their own reading of the source code.

Autotest understands each assignment might consist of parts, so for example, if the source files required for one part are missing or won't compile, the other part will still be tested.

What's new

The first version of Autotest was a framework that required the instructor to compose a Java class to describe each assignment. Frameworks are a great way to get 80% of the benefit for 20% of the work. The framework gave the instructor a lot of flexibility without requiring me to anticipate everything they might need. The disadvantage is that writing classes is a kind of grody and error prone way to customize something. Autotest 2 is a full-fledged application with a GUI editor for assignment descriptions. It can even be used by someone who doesn't know Java. In order to retain much of the flexibility of the framework approach, Autotest 2 uses scripting. For example the reports sent to students are created by executing a script.

What are the main assumptions?

Files submitted by students must each be in a separate subdirectory. We use the student's login name as the directory name, but it doesn't have to be so. All these subdirectories are in one directory called the submit directory.

Autotest will copy all the files submitted by one student from their submit subdirectory to a second directory called the working directory. The files are then typically compiled and, when necessary, linked with the test bench code. Then a number of test commands are issued. By default the test command is "atest i", where i is the number of the test, but the user has full control of what command is used. These commands run in a separate process. The exit code of that process determines whether the test is considered to have passed (0) or failed (not 0). Output from the test process is collected so it can be reported to the student.

Platform

Autotest 2.0 itself is written in Java and requires JDK 1.3 or higher. It is intended to be used in Windows NT, Windows 2000, Windows XP, Unix, or Linux. It might work in other environments such as the Windows 95 family or Macintosh OS X.

Scripting

A few aspects of Autotest require the use of a simple scripting language. In particular the reports sent to the student will be produced by executing a script, and any commands sent to the operating system are preprocessed by first running them through the scripting engine. Scripts are processed within an environment which is a mapping from variable names to values. The environments used are described in this document. The scripting language is described in a companion document: Simple Script Help.

Sending Commands to the OS

To do a lot of its work, Autotest "sends commands to the OS". Exactly what this means depends a bit on the platform. Autotest uses Java's RunTime.exec(String, String[], File) method to do this. In an ideal world you could look up the documentation for RunTime.exec(String, String[], File) on your platform; good luck finding it.

Typically the commands are interpreted just as if you had typed them into a Unix shell or a Window's "Command Prompt" window. But there are some differences; shells support piping and file redirection; RunTime.exec (and hence Autotest) does not. Shells support some commands (e.g., "cd") directly, and so these can not be used with Autotest. Unix shells expand wildcards, so wildcards can't be used by Autotest in Unix. In general if you want to do something fancy like piping write a shell script (Unix) or a .BAT file (Windows) and use that file's name in Autotest.

Commands are parsed using space characters to separate the arguments from the command name and from each other. Arguments with spaces in them should be double-quoted. Currently there is no way to deal with arguments that have double quotes in them.

Commands are provided with no input on their standard input stream. Output to standard output and standard error streams is captured by AutoTest.

Commands are first run through the scripting engine. Then each non-blank line is sent to the OS until either a command fails or all commands have been sent. A command fails if it has a nonzero exit value, crashes, or times-out. Therefore it is important to know what your commands return for an exit value. For example if you turn warnings level to full on a compiler, then will it return a nonzero exit code when it merely issues a warning?

For Windows users:

In Windows I noticed that executable files that I created in the working directory required an absolute path name. So I recommend using absolute path names (the scripting variable workingDir will contain the drive letter and directory part of the absolute path name).
To execute a .BAT file in Windows, give its full name. You can omit .EXE, but not .BAT.
The CMD command can be used to access commands normally executed by the shell. For example:

CMD /c ERASE /q foo.class

If you use Microsoft's Developer Studio, the command CL -GX seems to work well for running the compiler (use the -c option to suppress linking). Of course you have to make sure that the directory containing CL.EXE is on your PATH environment variable.

Assignment descriptions

Assignments descriptions are kept in files. The recommended suffix for these files is .at2.

To facilitate editing these files, there a GUI editor lets you edit these files. The following fields describe an assignment.

Course Name: This can be whatever you want. It is used only in reporting.
Assignment Name: This can be whatever you want. It is used only in reporting.
Submit Dir: This is the name of the directory under which student submitted files are found. Within this directory there should be one directory for each student. Within those directories the files submitted by the students should be located prior to running the assignment. It is recommended to use the full (absolute) path name for this directory (including drive letter, in Windows).
Marked Dir: This is the name of the directory under which reports should be placed. During a run, Autotest will create a directory for each student within this directory and within that directory it will place the report file and the output file. It is recommended to use the full (absolute) path name for this directory (including drive letter, in Windows).
Working Dir: This is the name of the directory where all the work takes place. Student submitted files are copied here. Object and executable files will usually be created here. The test bench will typically be placed here. It is recommended to use the full (absolute) path name for this directory (including drive letter, in Windows).
Report file: This is the name of the report file created for each student.
Output file: This is the name of the output file created for each student.
Initial Compilation Commands.: The commands here are used to set up the test bench. Typically I use it to compile my test.cpp file to an object file. During a run of an assignment, the initial compilation commands are first run through the script engine using as environment that contains information about the assignment. (See Environments.) They are then sent to the OS.
Report Script: This script is used to compose the report file for each student. The environment is discussed later. (See Environments.)
Output Script: This script is run to produce the output file that is sent to the student. Typically this is where Autotest will put the output captured from the tests, whereas the report file contains just the bare bones pass/fail information. The environment used for this script is the same as that used for the report; so you can assign to variables while writing the report and use those variables while writing the output. (See Environments.)

Part Descriptions

Each assignment is assumed to be made up of 1 to 10 parts. For simple assignments, just use 1 part. The defining characteristic of a part is that either all or none of the tests for a part will be run. Unless all files for a part are present and compilation for the part succeeds, no tests will be run for that part.

Each part is described by the following fields.

Part Name: This can be whatever you want, it is just used for reporting.
Submitted files: These are the names of the files the student should have submitted. Entered one per line.
Compilation Commands: These commands are sent to the OS after the student files are copied. Typically I use them to compile the submitted code and to link it with the test bench. During a run of an assignment, the compilation commands are first run through the script engine using as environment that contains information about the assignment and the part. (See Environments.)
Number of tests: This is the number of tests for the part.
Double Check: If this check-box is checked, then a command will be run to double check that the number of tests that is in the part description is the same as what the test bench is prepared to handle. To query the test bench a command is sent to the OS and the exit code of that command should be the number of tests. For my test bench the command is "atest count".
Test Commands: Each Test Command is a line to be sent to the OS. The test is considered passed if the program exits normally and with an exit code of 0. For my test bench the command is "atest t" where t is the number of the test. Test commands are scripted. (See Environments.)
Capture stdout and stderr.: This indicates when captured output will be reported to the student. All this really does is control the value of the testCapturedOutput[p][t] script variable. If output is not captured this value will be the empty string.

Running an assignment

To run an assignment:

select the Run menu item, or click on the running person icon,
select the students you want to test,
click on go.

This initiates the following algorithm (ignoring the many things that can go wrong)

If required run the initial compilation commands
For each selected student
- For each part
  - Copy the students files to the working directory
  - Run the part's compilation commands
  - Double check the test count
  - For each test
    - Run the test
- Write the report and output files for the student.

Scripting Environments

For assignment-level scripts you can use the following variables:

courseName	The course name
assignmentName	The assignment name
workingDir	The working directory (absolute path)
submitDir	The submit directory (absolute path)
markedDir	The marked directory (absolute path)
partName	A vector of part names

For part-level scripts the following variable is also available

partNum

The number of the part (counted from 0)

For tests you have in addition:

testNum

The number of the test, counting from 0.

For the report file and the output file, you have the following variables in addition to those for the assignment.

studentName	The directory name for the student.
partCompiled	A vector of numbers, one for each part: 0 if a file wasn't submitted, 0 if compilation failed, and 1 otherwise.
partErrors	A vector a strings, one for each part: "" if the part compiled and otherwise a message indicating the nature of the problem.
testScores	A vector of vectors of numbers. testScores[p][t] is 0 if test t of part p was failed and 1 otherwise. When partCompiled[p] is 0, testScores[p] will be an empty vector. (Note the average of an empty vector is 0.0.).
testCapturedOutput	A vector of vectors of strings. testCapturedOutput[p][t] is the output from test t of part p. When partCompiled[p] is 0, testCapturedOutput[p] will be an empty vector.

To calculate the average over the parts you can use

    { average for ps in testScores do average ps end }

To calculate the average over all tests you SHOULD NOT use the script

    { sum (for ps in testScores do count ps end)
      /  sum (for ps in testScores do # ps end)}

since this will only count tests for parts where compilation was successful. At the moment, the only way to do this seems to be to manually count the number of tests. For example

    { sum (for partScores in testScores do count partScores end) / 13 }

How to write a test command

That's entirely up to you, as long as the test command exits with 0 when the test is passed and exits with something else when the test fails.

C++

I have a generic unit test program called atest.cpp that handles argument parsing and handles exceptions. This program calls two subroutines, which are custom written for each part.

int testCount() ;

and

void doTest(int, int &) ;

The first just returns the test count. The second takes the number of the test and computes a result (0 means pass). The doTest subroutine is typically implemented as a big switch statement, with each branch executing a simple unit test, and usually outputting some useful information, about what the expected and received outputs of the unit were. See the examples that come with the software.

Where the students are required to write standalone batch programs, the tests can be shell (or BAT) files that pipe the input into the program and do a diff (or something more clever) on the output. Dennis Peters had a student working on a system for automating the testing of stand-alone programs.

Java

For stand-alone programs, you could use Dennis's system mentioned above, or obtain a test driver that I (Theodore Norvell) wrote. Mine uses regular expressions to check the output.

For unit tests, you can roll your own, or capitalize on JUnit. The following test command works fine.

    java -ea -cp ".;C:\...\junit.jar" junit.textui.TestRunner TestClassName

For GUI programs, life gets more interesting.