Testing

Overview

This describes features in Planimate 8.4 with the primary purpose being to enable the automatic verification that changes made to Planimate do not create unexpected changes in the run results of existing simulation models.

Benefits for Planimate development

• Improving the quality of releases by having access to non trivial models which use the simulator in varied ways.
• Less distraction to modellers for testing in exchange for an investment by modellers in incorporating tests into their models.
• A resource that can guide decisions on Planimate development based on which features are detected in use most often
• Test cases for profiling and optimisation planning, and for testing optimisation attempts.

Benefits for Planimate developers:

• A mechanism developers can use to test new versions of Planimate
• A component that developers can use in conjunction with their own model development and debugging processes (which are beyond the scope here at this time).

Key assumptions

• The Planimate test suite collection will comprise primarily of finished and delivered models. Its not intended to assist early stages of model debugging. This will be open to discussion case by case.
• This is not a replacement for module/unit testing and modellers should not submit such tests to the test suite, the focus is in complete runs as would be experienced by the end user.
• Internally modellers may use the new Planimate capabilities as they see fit.
• The total time that a given model needs for its test needs to be “reasonable”. Proposed 10-30 minute total limit for all subtests of a given model? [TBD]
• The focus is currently on testing simulation as the automation of user interaction is a separate issue beyond the scope at this stage.
• Models that are not actively maintained/kept up to date may have to be dropped from the test suite as Planimate develops beyond their compatibility.

The development of this involves both enhancements to Planimate and development/adoption of a test management system. This document focusses on the principles and Planimate enhancements to support them.

Basic Principle

Modellers add a handler to a model to enable it to run in batch mode with the primary result being to set a system attribute if an expected result is achieved. In detail:

1. The test suite will run models in Planimate in batch mode. A standalone (outside test suite) run will be possible so modellers can use these mechanisms for their own testing if they wish.
2. Modellers submit a model, and optionally a TEST.csv testing table file and associated dataset (ds2) files.
3. The testing suite does not need DB/map tile files as it does not load images.
4. Upon load the test model is automatically run. Instead of _Model_Loaded, a _Test_Start broadcast is sent as well as s.TestNumber being set non zero.Both indicate testing mode and the modeller can choose how they detect testing mode, what is important is to run the simulation without need for user intervention from when the _Run_Start broadcast is sent.
5. A data set filename can be provided for the model, the model reads this using s.DataFileNamePath text attribute.
6. When the simulation finishes (time reached, no more events, modeller triggers a stop) Planimate will automatically close down, in that process another new broadcast, _Test_Finish is sent. During this time (or earlier if the modeller prefers) the system attribute s.TestResult needs to be set to signify a successful or failed test.
7. A new system attribute s.TestTarget can be used to pass a text string to the model (eg: a checksum). If a modeller implements an internal test target, this need not be used.
8. The modeller does not have to worry about arranging advance to time. The batch run takes care of this.
9. The model should not expect or try to save itself, Save On Exit will be ignored.

Related Constructs

BC “_Test_Start”	Sent first to model instead of “_Model_Loaded” when model is run with with the /TESTRUN command line option by the testing system. The model can use this to prepare for testing.
BC “_Test_Finish”	Sent to a model after the simulation has stopped/paused/ended, just before Planimate exits. This is the last chance for the modeller to set s.TestResult, it can be set at any time earlier. The demonstration test portal uses this broadcast.
s.TestNumber	Set to 1 when /TESTRUN is used to run Planimate. If /TESTRUN=x is used, set to X, X must be >= 1. This both indicates a test and the test number. A modeller could ignore the “_Test_Start” broadcast and check this instead.
s.TestResult	The model must set this to indicate success or failure. Not setting it implies a failure.
s.TestTarget	This text system attribute is set by the test suite using the /TESTTARGET=”” command line option and would typically contain an expected hash for the test to compare against. If not set, it is an empty string.
s.LoadFileNamePath	This text system attribute is set by the test suite using the /DATASET=”” command line option. The model can use this to load a dataset based on rows in the TEST.CSV file described below.
TEST.CSV	A modeller provides this along with a model to define multiple runs. If present, the test suite will run the model for every row, passing the dataset, expected target and row/test number (the system attributes described above). The model under test may read TEST.csv for additional parameters, it can assume the file will be in its directory.
l._test_results	This is a label list with the following values: 0: “-” unset (test failure) 1: “Success” (test OK) 2: “Fail” (test failure)
LogMessage()	This routine operation should be used to report why a test failed. If a checksum failed, log the calculated text as follows: “CalculatedHash:XXXXXXXXXXXXXX” The platform logs the TestTarget and TestNumber for you. Log any other comparisons which cause a test to fail.

Multiple Test Support

A modeller might want to test a model under a number of configurations, using a number of datasets, random seeds etc. The TEST.CSV file enables this.

The file is to be CSV with double quoted text fields and contains the following columns:

column 1: data set name (text, no paths)column 2: expected result eg: hash for SHAFile() (text)column 3,4 etc are optional extra column, as needed by modeller, if text, its to be quoted.

For each row in this table Planimate is invoked as follows:

planimate.exe /TESTRUN=<row number> /DATASET=<dataset name> /TESTTARGET=<expected result> (… other internal stuff like /batch)

/TESTRUN sets s.TestNumber to >= 1 and activates the testing features./DATASET sets s.LoadFileNamePath, modeller can load scenario etc./TESTTARGET sets s.TestTarget which can be used to pass a hash or other test result information.

Meaning of Success

The recommended and least-impact way of testing the “success' of a run is for a model to write key KPIs or tables to a file and compute a checksum for the file using SHAFile(). This approach has been called Dump And Hash.

Dump And Hash

By giving tables data object labels, a few lines of routine code can easily iterate through the tables and create a CSV format file with the contents of all the tables. Sample code will be provided which includes labelling the tables and their location in the model.

Once this file is created, the FileSHA() operation can be used to create a 64 hex character hash which will detect any difference in the CSV file. Such a difference can then be used to flag a failure. The expected hash will be provided to the model by s.TestTarget and is readily viewed and updated in TEST.CSV.

For best results, please follow the following guidelines:

• Write the file that will be hashed in CSV format and using “Formatted” as raw numbers differ in the 15^th decimal place (refer to reference module for preferred options). This also has the benefit of exercising label,text handling, value formatting and calendar handling code as part of the test.
• Calculate the hash using FileSHA() with the “Text File” option set so that line endings on different platforms do not impact the test.
• Don't delete the file after the test, it may be useful for further processing when a test fails
• Use LogMessage() to log a message of the form “CalculatedHash:XXXXXXXXXXXXXX”. The sample module will show the string expression that does this simply.
• Don't forget to set s.TestResult

IMPORTANT

The test mechanism will not be accessible for PBAs for obvious security reasons. This minimised impact on models and the test code can stay resident even in a MDL that is to be made a PBA.

Bundling Data Sets

A test model can be bundled with one or more dataset files. These should be named with a name that associates them with the specific model.

TestModelA_1.0.MDL

TestModelA_1.0_DATA1.DS2

TestModelA_1.0_DATA2.DS2

The aforementioned TEST.CSV file associates the model with the datasets and expected results.

An optional use mode is to not bundle a TEST.CSV, in this case the model is invoked once, s.TestNumber will be 1 and both s.LoadFileNamePath and s.TestTarget will be empty. In this configuration it is assumed the model can do a self contained test. If it fails it is important that LogMessage() give useful detail why.

File Handling

Models under test should reference data sets and test configuration files without directory/paths. They can assume they have write access in their current directory and datasets will be in their directory. If the model creates a file to hash, leave the file (as described previously). Other temporary files can be created/deleted as long as absolute paths are not used.

The use of GetMyDocumentsName() and GetAppDataName() is supported. Under a test run, these will point to directories in the test staging area, effectively sandboxing any changes made within.

A test model cannot assume persistence of any settings it saves (including itself) and must ensure it can run (eg: not expecting a key file). Model code can test s.TestNumber being non zero to avoid key file checks etc. End users cannot set this for a PBA EXE.