Auto-tester

Preparing for Auto-testing

1. Download the zip file containing all the necessary Autotester files. It has these files:
   • AutoTesterInUse.sln, AutoTesterInUse.suo, AutoTesterInUse.ncb, AutoTesterInUse.vcproj, Release/AutoTesterInUse.res
     Visual Studio.Net 2008 related files. Simply click on the sln file to open the solution in your Visual Studio IDE.
   • AutoTesterLib.lib
     This is a static library containing the object (.obj) files that will be subsequently linked with the object files of your SPA. This one works under the "Release" mode, when you build your solution.
   • AutoTesterLib_debug.lib
     This is a static library containing the object (.obj) files that will be subsequently linked with the object files of your SPA. This one works under the "Debug" mode, when you build your solution.
   • AbstractWrapper.h
     This is a header file that must be placed in the same directory as TestWrapper.h. Please do NOT modify this file (otherwise, it will cause problems for the AutoTester later).
   • TestWrapper.h
     This is a header file containing the specification of a TestWrapper class that is in charge of the communication between the Auto-tester and your SPA.
   • TestWrapper.cpp
     This file will contain the implementation of the various methods of the TestWrapper class. The implementation of these methods will be done by you. Comments are included in the file to guide you.
   • Release/trial.txt
     A sample source program (in SIMPLE) for you to test your TestWrapper implementation.
   • Release/trialcase.txt
     A set of queries to go with the sample source program. Note that this is just a sample set of queries. The source program and queries used in the final testing will be more complicated than this ! Also note that this set of test queries may include queries that you have not implemented yet. Please modify accordingly. The first line of this file contains the total number of queries in the file. Use 'cat : <category>' to categorize your test cases (<category> will be displayed in the result). Rest of the file specify test cases - five line for each case: test case id, declaration, query, expected result, and allowed time in milliseconds.
   • Release/trialcase_NEW.txt
     The same queries as those in 'trialcase.txt'. This test case file contains the category tags and comments for each query inside.
   • Release/analysis.xsl
     After full run of test cases, the auto-tester will generate an xml file (out.xml). Open this xml file in Internet Explorer. IE will automatically convert the xml file to an html file using analysis.xsl.
2. To hook up to the Auto-tester, you only need to implement the TestWrapper class. The following is the source of TestWrapper.h:

   #ifndef TESTWRAPPER_H
   #define TESTWRAPPER_H
   
   #include <string>
   #include <iostream>
   #include <list>
   #include "AbstractWrapper.h"
   
   // include your other headers here
   
   class TestWrapper : public AbstractWrapper {
   public:
   // a default constructor
   TestWrapper();
   
   // method for parsing the SIMPLE source
   virtual void parse(std::string filename);
   
   // method for evaluating a query
   virtual void evaluate(std::string query, std::list<std::string>& results);
   };
   
   #endif
   

   First, in the header file (TestWrapper.h), include any other header files necessary for compiling the TestWrapper class. If you need to use instance variables in your implementation, create a private section in the class specification and declare your instance variables there. Next, you need to implement the 3 methods specified in TestWrapper.h by writing code for the methods in TestWrapper.cpp.

   The first method is the default constructor. In this method, you can create objects that you will need later on for parsing and query evaluation. You can also do any initialization your need for your SPA here. The second method is called parse. The Auto-tester will pass the name of the SIMPLE source file through parameter "filename" of this method. What you need to do here is to write code that will invoke the parser/design extractor of your SPA to create the PKB components from the source file.

   The third method is called evaluate. It has 2 parameters: query will contain the query string while results will store the answers to the query. Here, you will write code that will invoke your SPA to evaluate the query. The answers to the query will be subsequently stored in the results list.

   The PQL query comes in the form of a single line (Even though it is given as two lines in the test case file - those two lines are concatenated by the Auto-tester with a space in the middle). For example, "assign a; stmt s; Select <a, s> such that Parent(s, a)". There's only a space between the declarations and the query (not a newline). On the other hand, answers to the query must be stored as std::strings in the results list. For example, if the answer to the query is 1 2, 1 5 and 1 8, results will contain 3 elements: "1 2", "1 5" and "1 8". We shall now discuss the different formats of the returned results:

   • Select BOOLEAN ......
     For boolean results, the results list contains only one element - "true" or "false", with all characters in lowercase.
   • Select a such that ....
     For example, if your answer is 9, 7, 1 and 3, then the results list will contain 4 string elements: "9", "7", "1" and "3". Note that you only need to return the statement number(s) if the synonym in the Select clause is of type statement, assign, while or if. If the synonym is of type variable or procedure, return the variable and procedure names respectively.
   • Select <p, s> such that ...
     For tuple results, each result is stored as a string in the results list. However, each component of a result must be separated by one space. For example, if the results are <p, 1>, <q, 4>, then the results list will contain 2 elements: "p 1" and "q 4". Notice that there is only one space between p and 1 as well as q and 4 (two or more spaces are not allowed).
   • Select s.stmt#... or Select <p.procName, v.varName>...
     The format of the results is similar to the previous two cases. For example, if the results to "Select s.stmt#..." are 1 and 6, the results list will contain 2 elements: "1" and "6".
   • If there are no results, just leave the results list empty. Don't add anything to the results list, not even an empty string.

   Important: The order of the results is not important. The Auto-tester will do the sorting of the results for you. The Auto-tester will also remove any duplicates it finds in the answers. For example, if you return the answers "1", "3", "3", "2", "3", the Auto-tester will consider your answers to be just "1", "2" and "3". Note also that when the result is a procedure name or variable name, make sure you do not change the case. For example, if the result is procedure First, it should be returned as "First", not "first" or "FIRST" or "FiRSt" etc.

3. The Auto-tester works in a cooperative mode. In other words, instead of forcing your program to stop whenever the time limit is reached during the evaluation of a query, it will first signal to your program to stop. Then, it waits for a couple of seconds for your program to do any clean up e.g. deallocating memory etc. If your program returns within the extended time limit, the Auto-tester will continue with the next query. Otherwise, it will print out a message to inform us that the program has refused to terminate, in which case we will forcefully terminate the whole testing and give a full penalty for that query.

   It is important that you return the goodwill of the Auto-tester by responding to the stop signal. We will heavily penalize programs that refuse to cooperate with the Auto-tester. To do the checking of the stop signal, you need to add some code into your own SPA program as sketched out in the following pseudocode:

   	#include "AbstractWrapper.h"
   
   	.....
   
   	for each query_clause in query do
   	  evaluator.eval(query_clause)
   	  if (AbstractWrapper::GlobalStop) {
   	    // do cleanup 
   	    ....
   	    return
             }
   
   	.....
   	

   First, ensure that you add the AbstractWrapper header file Thread.h using '#include "AbstractWrapper.h"'. To check for the stop signal, you check the static global variable AbstractWrapper::GlobalStop which is a bool value. If the value is true, it means that the time has expired for the query and you should stop execution and do any cleanup and return control back to the Auto-tester. Otherwise, you are still within the time limit and you can continue processing the query.

   You can do the checking at more than one place in your program. For example, the evaluation of Affects* is often time consuming. So, you might want to do a check for every 50 answers found during the evaluation of an Affects*. Since every team's implementation is different, you have to judge for yourself where are good points in your program to do this checking. Generally, you will want to do checking in strategic places e.g. after the evaluation of a query clause. However, note that checking takes time. Therefore, too much checking may slow down your system. There is no hard and fast rule on how much checking is required. You have to find a suitable level that will strike a balance between the efficiency and cooperativeness of your implementation.

Compiling and running the Auto-tester

Important: The `main' method is located in the Auto-tester. Therefore, you should not compile a source file that contains the method main. Either remove it or comment it out.

To run the Auto-tester, enter the following command from the command prompt at the release directory:

AutoTester trial.txt trialcase.txt out.xml

Once the testing is completed, you can view the results of the testing by opening out.xml in Internet Explorer (IE); IE will be using analysis.xsl to convert out.xml to an HTML document.

Important points

There are some important points you need to take note of:

1. The Auto-tester program make use of the namespace AutoTester. Therefore, if you are using namespace in your program, you should avoid calling your namespace AutoTester.
2. Please follow the answer format strictly. For example, if you're returning tuple results such as "1 2", "3 4" etc, make sure that you don't have excessive spaces in-between. Answers such as "1     2" will be regarded as wrong by the Auto-tester. This will in turn result in unnecessary time waste when we double-check the failed cases.
3. During the implementation of your TestWrapper, you might want to print some debugging information to the standard output. This is allowed but please remove/comment out those std::cout statements before you go for the final testing. Printing debugging information during the testing will slow down your SPA and result in unnecessary timeouts (each query is only allowed a certain amount of time to complete)
4. Please compile your program in Release configuration. By default, AutoTesterInUse solution uses Release configuration. Mostly due to inappropriate coding practice, one might find problem/bugs when switching between Debug and Release configuration. Release configuration performs optimization, while Debug doesn't. For a list of advice on this issue see:
   • http://www.flounder.com/debug_release.htm,
   • http://www.codeproject.com/debug/releasemode.asp?df=100&forumid=178&exp=0&select=904#xx904xx.
5. The SIMPLE source file used for the final testing will be around 300 to 400 lines. The number of test queries are also around 300-400. So, do be prepared for an extensive final testing. You should also try to run your own SPA for a large set of consecutive queries say 100-200 queries on a specific SIMPLE source file to see whether your SPA is robust enough to withstand large number of queries without exiting
6. Auto-tester is not 'open-source'. Please do not request for the source code for Auto-tester.
7. You are welcome to use Auto-tester for your own system testing. But the main purpose of Auto-tester is final testing, for which it has been proved to be adequate. Therefore, no requests for enhancements please.