Sim Tester - TestCase Objects

A TestCase object is the building block of a test suite, and is itself composed of the actual tests that you wish to run.

Test Execution

Every time JUnit runs a test, it can be thought of as running the following Java code, which runs until the test fails (an AssertionFailedException is thrown) or an unexpected error occurs (any other type of Exception is thrown):

TestCase testCase = ...

try
{
   testCase.setUp();
   testCase.testSomething();
   testCase.tearDown();

   // the test passed!
}
catch(AssertionFailedException e)
{
   // the test failed because of an expected error
   // (i.e. some assertTrue(), assertEqual(), etc. failed)
}
catch(Exception e)
{
   // the test failed because of an unexpected error
   // (e.g. a null pointer, a network error, ...)
}

The first thing that happens is the calling of setUp(). This method should do all of the initialization work that has to happen for each test (note that the constructor is usually not used, since it only executes when the TestCase is instantiated, not once per test). If there is a problem at this stage (e.g. the ConnectionManager is unable to get into the required STEAL state), an Exception will be thrown, and the test is reported to have encountered an error.

If setUp() succeeds, the test method is run. Tests typically consist of:

  1. Sending STEAL strings to the simulator (via Connection's sendLine() method)
  2. Parsing STEAL responses (via Connection objects, using Response objects)
  3. Checking returned values (using Response objects and assertion methods)

If the test completes successfully, tearDown() is called, which may be required to clean up data structures, close connections, etc.

Responses

There are only so many valid responses that can come back from a simulator. Rather than parsing these responses with each test, we have created Response classes that parse (and validate) responses for you.

When you expect, for example, an error response, simply construct a new ErrorResponse object, passing it the Connection as a parameter:

connection.sendLine("xxxxxxxx");
new ErrorResponse(connection);

The Response will ensure that a response is received within .1s, and that it conforms to an expected pattern. Depending on the response, it may also parse values and provide methods for accessing them. The types of response are:

Helpful TestCase Methods

junit.framework.TestCase and ctf.evaluation.TestCase provide several methods that you may find useful during your testing.

public static void assertEqual(T expected, T actual) [JUnit]

This method tests whether or not a value is what you expect it to be. The method is implemented for many types, including numbers, strings and even plain Object. There are also versions of assertEqual() that accept message strings, which are reported if the assertion fails.

Be careful using this method with floating-point numbers: sometimes 5.0 is not equal to 5.0 (see assertClose(), below().

public static void assertTrue(boolean) [JUnit]

Similarly, assertTrue() and assertFalse() allow you to check whether or not a boolean value (e.g. "can I see the enemy's flag?") is what it ought to be. There are also versions which take message parameters:

public static void assertClose(double expected, double actual) [Sim Tester]

Because of the way that floating-point numbers are represented by computers, two numbers that look alike may not actually be (e.g. 5.0 may actually be stored as 4.99999999 or 5.00000001). Building tests using assertEqual() can lead to error messages such as "Assertion failed: value was 5.0, expected it to be 5.0". The variant of assertEqual(String, double, double, double) takes a tolerance as its last argument for exactly this reason. ctf.evaluation.TestCase adds the method assertClose(), which checks to see if two values are within a percentage tolerance. This is mostly here for legacy reasons.

This method has several variants:

  1. public static void assertClose(double expected, double actual)
  2. public static void assertClose(double expected, double actual, double tolerance)
  3. public static void assertClose(String message, double expected, double actual)
  4. public static void assertClose(String message, double expected, double actual, double tolerance)

Each checks to see if two values are close. For instance, if the tolerance value is .001 (the default value, used in variants #1 and #3), we check to see that neither is more than 0.1% greater than the other. Variants #3 and #4 allow you to specify the error message if the assertion fails, and #2 and #4 allow you to specify the tolerance.

protected void sleep(int milliseconds) [Sim Tester]

This method suspends execution for the specified number of milliseconds (using Thread.sleep()), catching java.lang.InterruptedException so that you don't clutter up your test with try/catch blocks.