Testing

AUTOMATED TESTING

The automated testing was designed to catch errors that may occur when making changes to the core simulation code or introducing new methods of simulating a circuit. At the core of the system are seven standard test circuits. They are listed below. Each circuit can be any size and is made up of a single, basic building block that is repeated throughout the circuit. The circuits were designed to produce a single definite output from a single input. They were designed to catch problems that were likely to occur whilst being easy to verify.

1. SQRTNot test, standard build. This test involves two sequential square root of not gates, the combined effect is to invert the input. This circuit is built using pin connection (i.e. connect gate 2, output 3 to gate 7, input 1) logic, which may involve out of order connections.
2. SQRTNot test, bit build. As above, but with the connections specified using bit connection logic (i.e. connect gate 7 input 1 to the last gate to output bit 3), which must be specified in order.
3. Flip test, standard build. This test uses two sequential 'Flip' gates. The combined effect is to invert selected bits only if certain other bits are set.
4. Flip test, bit build. As above, but specified using bit connection logic.
5. Cascade Flip test. Two sequential Flip gates, cascading down from the top left, as shown in Figure 33 (a)
6. Reverse Cascade Flip test. Two sequential Flip gates, cascading up from the bottom left, as shown in Figure 33 (b)
7. Not-Flip-Not. Test. Eight square root of not gates and two Flip gates, arranged as in Figure 33 (c)

_{Figure 33}

The first four tests were designed to highlight any problems in the basic creation of the circuits and the simulation of simple systems. The last three tests were designed to test the interactions between bits and catch problems that may come from attempting to order the calculations.

The automated tests selects a test circuit randomly, makes it a random size (up to user a specified maximum), chooses a random input, selects a random simulation algorithm (simple or complex), selects a random number of threads if the algorithm is complex and then randomly chooses between static and sparse vectors. The combined effect is to allow hundreds of millions of possible tests.

As the automated test algorithm knows which test it has created, and the inputs to that test, it is able to work out what the output should have been. Upon the test's completion the output is compared to the ideal and the opportunity to stop the program is given to the user. The automated tests continue until the user intervenes.

I have not yet found a circuit that fails to simulate correctly after the simulator has passed a large number of these tests. Typically, major errors in the simulator implementation are revealed quite early by these tests. Minor problems, such as problems associated with accessing memory via C++ pointers, usually appear within a hundred tests or so. I will typically run 2,000 tests after changing the simulator code and periodically run 10,000 tests.

INTERNAL TESTS

The debug version of the simulator has over 200 testing points scattered throughout the code where pre-conditions, post-conditions and invariants are tested throughout execution. The application is very stable and has few known weaknesses. Exception handlers are used in the program so that an error in most parts of the program will be handled gracefully and not cause the entire application to crash.