It's newer and guier.
Autotest is a system for automating some of the tasks involved in testing software assignments. Specifically Autotest will
There is a lot that can go wrong in the above process and Autotest is capable of dealing with
There are a number of tasks that you might think Autotest ought to do but doesn't.
Autotest understands each assignment might consist of parts, so for example, if the source files required for one part are missing or won't compile, the other part will still be tested.
The first version of Autotest was a framework that required the instructor to compose a Java class to describe each assignment. Frameworks are a great way to get 80% of the benefit for 20% of the work. The framework gave the instructor a lot of flexibility without requiring me to anticipate everything they might need. The disadvantage is that writing classes is a kind of grody and error prone way to customize something. Autotest 2 is a full-fledged application with a GUI editor for assignment descriptions. It can even be used by someone who doesn't know Java. In order to retain much of the flexibility of the framework approach, Autotest 2 uses scripting. For example the reports sent to students are created by executing a script.
Files submitted by students must each be in a separate subdirectory. We use the student's login name as the directory name, but it doesn't have to be so. All these subdirectories are in one directory called the submit directory.
Autotest will copy all the files submitted by one student from their submit subdirectory to a second directory called the working directory. The files are then typically compiled and, when necessary, linked with the test bench code. Then a number of test commands are issued. By default the test command is "atest i", where i is the number of the test, but the user has full control of what command is used. These commands run in a separate process. The exit code of that process determines whether the test is considered to have passed (0) or failed (not 0). Output from the test process is collected so it can be reported to the student.
Autotest 2.0 itself is written in Java and requires JDK 1.3 or higher. It is intended to be used in Windows NT, Windows 2000, Windows XP, Unix, or Linux. It might work in other environments such as the Windows 95 family or Macintosh OS X.
A few aspects of Autotest require the use of a simple scripting language. In particular the reports sent to the student will be produced by executing a script, and any commands sent to the operating system are preprocessed by first running them through the scripting engine. Scripts are processed within an environment which is a mapping from variable names to values. The environments used are described in this document. The scripting language is described in a companion document: Simple Script Help.
To do a lot of its work, Autotest "sends commands to the OS". Exactly what this means depends a bit on the platform. Autotest uses Java's RunTime.exec(String, String[], File) method to do this. In an ideal world you could look up the documentation for RunTime.exec(String, String[], File) on your platform; good luck finding it.
Typically the commands are interpreted just as if you had typed them into a Unix shell or a Window's "Command Prompt" window. But there are some differences; shells support piping and file redirection; RunTime.exec (and hence Autotest) does not. Shells support some commands (e.g., "cd") directly, and so these can not be used with Autotest. Unix shells expand wildcards, so wildcards can't be used by Autotest in Unix. In general if you want to do something fancy like piping write a shell script (Unix) or a .BAT file (Windows) and use that file's name in Autotest.
Commands are parsed using space characters to separate the arguments from the command name and from each other. Arguments with spaces in them should be double-quoted. Currently there is no way to deal with arguments that have double quotes in them.
Commands are provided with no input on their standard input stream. Output to standard output and standard error streams is captured by AutoTest.
Commands are first run through the scripting engine. Then each non-blank line is sent to the OS until either a command fails or all commands have been sent. A command fails if it has a nonzero exit value, crashes, or times-out. Therefore it is important to know what your commands return for an exit value. For example if you turn warnings level to full on a compiler, then will it return a nonzero exit code when it merely issues a warning?
For Windows users:
CMD /c ERASE /q foo.class
Assignments descriptions are kept in files. The recommended suffix for these files is .at2.
To facilitate editing these files, there a GUI editor lets you edit these files. The following fields describe an assignment.
Each assignment is assumed to be made up of 1 to 10 parts. For simple assignments, just use 1 part. The defining characteristic of a part is that either all or none of the tests for a part will be run. Unless all files for a part are present and compilation for the part succeeds, no tests will be run for that part.
Each part is described by the following fields.
To run an assignment:
This initiates the following algorithm (ignoring the many things that can go wrong)
For assignment-level scripts you can use the following variables:
courseName | The course name |
assignmentName | The assignment name |
workingDir | The working directory (absolute path) |
submitDir | The submit directory (absolute path) |
markedDir | The marked directory (absolute path) |
partName | A vector of part names |
For part-level scripts the following variable is also available
partNum | The number of the part (counted from 0) |
For tests you have in addition:
testNum | The number of the test, counting from 0. |
For the report file and the output file, you have the following variables in addition to those for the assignment.
studentName | The directory name for the student. |
partCompiled | A vector of numbers, one for each part: 0 if a file wasn't submitted, 0 if compilation failed, and 1 otherwise. |
partErrors | A vector a strings, one for each part: "" if the part compiled and otherwise a message indicating the nature of the problem. |
testScores | A vector of vectors of numbers. testScores[p][t] is 0 if test t of part p was failed and 1 otherwise. When partCompiled[p] is 0, testScores[p] will be an empty vector. (Note the average of an empty vector is 0.0.). |
testCapturedOutput | A vector of vectors of strings. testCapturedOutput[p][t] is the output from test t of part p. When partCompiled[p] is 0, testCapturedOutput[p] will be an empty vector. |
To calculate the average over the parts you can use
{ average for ps in testScores do average ps end }
To calculate the average over all tests you SHOULD NOT use the script
{ sum (for ps in testScores do count ps end)
/ sum (for ps in testScores do # ps end)}
since this will only count tests for parts where compilation was successful. At the moment, the only way to do this seems to be to manually count the number of tests. For example
{ sum (for partScores in testScores do count partScores end) / 13 }
That's entirely up to you, as long as the test command exits with 0 when the test is passed and exits with something else when the test fails.
I have a generic unit test program called atest.cpp that handles argument parsing and handles exceptions. This program calls two subroutines, which are custom written for each part.
int testCount() ;
and
void doTest(int, int &) ;
The first just returns the test count. The second takes the number of the test and computes a result (0 means pass). The doTest subroutine is typically implemented as a big switch statement, with each branch executing a simple unit test, and usually outputting some useful information, about what the expected and received outputs of the unit were. See the examples that come with the software.
Where the students are required to write standalone batch programs, the tests can be shell (or BAT) files that pipe the input into the program and do a diff (or something more clever) on the output. Dennis Peters had a student working on a system for automating the testing of stand-alone programs.
For stand-alone programs, you could use Dennis's system mentioned above, or obtain a test driver that I (Theodore Norvell) wrote. Mine uses regular expressions to check the output.
For unit tests, you can roll your own, or capitalize on JUnit. The following test command works fine.
java -ea -cp ".;C:\...\junit.jar" junit.textui.TestRunner TestClassName
For GUI programs, life gets more interesting.