Sim Tester (Framework)

The Sim Tester is a CTF simulator testing framework, built on JUnit and the CTF Tools, meant to make system-level simulator testing easy. It also uses a technology called AspectJ, but you do not have to have AspectJ installed to develop and run tests.

The test architecture is based on JUnit. A TestSuite object contains a number of TestCase objects, which has public methods whose names begin in "test". Each of these methods is a test to be run by JUnit.

In our case, we have TestCase objects like Start and Version, which extend ctf.evaluation.TestCase, which extends junit.framework.TestCase (and adds a few useful methods). These TestCase objects are aggregated by the ctf.evaluation.simulator.SimulatorTests object, which is a junit.framework.TestSuite.

Adding a new TestCase (e.g. for another STEAL state) requires two steps: creating the TestCase and adding it to SimulatorTests.

The first step, creating the TestCase, can be accomplished by following this template:

package ctf.evaluation.simulator;

import java.io.IOException;
import ctf.network.*;

import ctf.evaluation.TestCase;
import ctf.evaluation.simulator.responses.*;
import static ctf.evaluation.simulator.ConnectionManager.STEALState.*;


/** Tests the STEAL state 'Foo' */
@SimulatorTest public class Foo extends TestCase
{
   /** Set up for a test */
   @Override public void setUp()
      throws IOException, NetworkException, ProtocolError
   {
      connections.enterState(START);

      red = connections.red();
      blue = connections.blue();
   }

   /** Tear down after a test */
   @Override public void tearDown() {}



   /** Test the ability of the simulator to do something. */
   public void testSomething()
      throws IOException, NetworkException
   {
      red.sendLine("xxxxxxxx");
      blue.sendLine("xxxxxxxx");

      assertClose(3.14159, (new XXXXResponse(red)).aValue());
      assertEqual(3, (new XXXXResponse(blue)).anotherValue());
   }


   /** Something else happens. */
   public void testDoSomethingElse()
      throws IOException, NetworkException, ProtocolError
   {
      blue.sendLine("xxxxxxxx");
      sleep(100);
      red.sendLine("xxxxxxxx");

      new ErrorResponse(blue);
      new ErrorResponse(red);
   }




   /** The connection manager */
   private ConnectionManager connections = ConnectionManager.instance();

   /** Actual connections to the simulator */
   private ctf.network.tcp.Connection red, blue;
}

The second step, modifying ctf.evaluation.simulator.SimulatorTests, looks like this:

// ...

public SimulatorTests()
   throws IOException, NetworkException, ProtocolError
{
   addTestSuite(ctf.evaluation.simulator.Start.class);
   addTestSuite(ctf.evaluation.simulator.Version.class);
   addTestSuite(ctf.evaluation.simulator.RequestSide.class);
   addTestSuite(ctf.evaluation.simulator.Something.class);
}

// ...

Once you've created the skeleton of a TestCase, you're ready to carry on with fleshing it out: adding actual tests.