Plug in matelo_TestStand

MaTeLo 5.5 TestStand plugin allows testers to automatically generate test suites and test cases in TestStand format in order to do Hardware in The Loop testing, from a MaTeLo usages Model. National Instruments TestStand is a ready-to-run test management for organizing, controlling and executing your automated test systems. TestStand allows you to develop and execute test sequences by calling test programs created in the programming language of your choice. When you execute a sequence, TestStand generates the test report that you can customize and edit in XML, HTML and ASCII formats.

How to generate TestStand test cases from a MaTeLo Model

A test executive is a program in which you organize and execute sequences of reusable code modules. Hereafter are described some concepts that are applicable to test executives in general and that you need to understand before using the MaTeLo interface:

Code Module:

A program module, such as a Microsoft Windows dynamic link library (.dll) or a National Instrument LabVIEW VI (.vi) containing one or more functions that perform a specific test or other action.

Step:

An individual element of a test sequence. A step may call code modules or perform other operations.

Sequence:

A series of steps you specify to execute in a particular order.

Subsequence:

A sequence called from another sequence. The subsequence is specified as a step in the calling sequence.

Sequence File:

A file containing the definition of one or more sequences.

TestStand Concepts

This section provides an overview of the TestStand concepts that you have to understand and that will be reused in the definition part of the test bench with MaTeLo.

Sequence Context:

A TestStand object that contains references to all global variables, all local variables and step properties in active sequences. The contents of the sequence context changes depending on the currently executing sequence and step.

To access a variable or property, a code module must have a reference to the sequence context.

Step Group:

A set of steps in a sequence. A sequence contains the following group of steps: Setup, Main and Cleanup. When TestStand executes a sequence, the steps in the Setup step group are executed first, the steps in the Main group come next, and at last the steps in the Cleanup group.

Module Adapters:

Most steps in a TestStand sequence invoke code in another sequence or in a code module. TestStand must know the type of code module, and how to call it. TestStand uses module adapters to obtain this knowledge.

MaTeLo interface with TestStand includes the following module adapters:

  • LabVIEW adapter – calls LabVIEW VIs with a variety of connector panes.
  • LabWINDOWS/CVI adapter – calls C functions with a variety of parameter types. The functions must be in DLLs.
  • C/C++ DLL adapter – calls functions or methods in a DLL with a variety of parameter types.
  • .NET adapter – calls methods or properties in a .NET assembly.
  • ActiveX – calls methods or properties in an ActiveX.
  • Sequence adapter – calls other TestStand sequences with parameters.

Step Types:

Just as a variable has a data type, each step has a step type. A step type defines the behaviour and properties of a step. TestStand includes a number of predefined step properties, and allows you to create your own custom step types.

MaTeLo interface with TestStand includes the following step types:

  • Action – use Action steps to call code modules that do not perform test but, instead, perform actions necessary for testing, such as initializing an instrument.
  • Pass/Fail Test – use a Pass/Fail test step to call a code module that makes its own pass/fail determination. After the code module has been executed, the Pass/Fail Test step type evaluates the Step.Result.PassFail property. If Step.Result.PassFail is True, the step type sets the step status to Passed. If Step.Result.PassFail is False, the step type sets the step status to Failed.
  • Numeric Limit Test – use a Numeric Limit Test step to call a code module that returns a single measurement value. After the code module has been executed, the Numeric Limit Test step type compares the measurement value to predefined limits. If the measurement value is within the bounds of the limits, the step type sets the step status to Passed. Otherwise, it sets the step status to Failed.
  • String Value Test – use a String Value Test step to call a code module that returns a string value. After the code module has been executed, the String Value Test step type compares the string that the step obtains to the string that the step expects to receive. If the string obtained by the step matches the expected string, the step type sets the step status to Passed. Otherwise, it sets the step status to Failed.
  • Custom – use Custom step types to call a generic step type that you created to fit the specific needs of your application. You can do this by modifying an existing TestStand built-in step type or creating a new one. You can by this way define the module adapter, the path to the code module, the parameters, etc. once and for all.

Parameters:

Code modules and sequences may have some parameters, used to pass data to and from it. The different ways to pass data to a code module in a step are the following:

  • Function Parameter – data is passed to the function through its prototype. If the step calls a subsequence, the behaviour is the same.
  • Step Parameter – each step in a sequence can have a set of properties, which is defined in the step type. A code module which has a reference to the TestStand sequence context can access data defined in step properties (as Step.InBuf).

[See TestStand Standard Step Properties to get a list of standard properties of the TestStand built-in steps.]

  • Local Variable – as local variables have properties that are local to a particular sequence, they are useful to store data relevant to the execution of the sequence. A code module which has a reference to the TestStand sequence context can access data defined in a local variable (as Locals.ResultList).
  • Global Variable – global variables have properties which are global to a sequence file, they are accessible by any sequence or step in the sequence file. A code module which has a reference to the TestStand sequence context can access data defined in a global variable (as FileGlobals.MyData).

 

Defining TestStand Functions and Sequences

 

This chapter describes the different ways you can use within Usage Model Editor to define the TestStand functions and sequences which are usable on your test bench.

By function, we mean a code module or a user-defined sequence that can be called in a step.

By sequence, we mean a set of references to functions which are grouped in a particular order, and that can be called on a step as a subsequence. The difference with a user-defined sequence is that a sequence in the sense of MaTeLo will be automatically created by MaTeLo when generating the test suite.

Within MaTeLo, a Test Bench is divided into three groups:

  • Setting group – contains functions used to perform actions necessary to initialize instruments or to set some properties on the SUT. These functions don’t perform any test, that’s why the only allowed step types for this group are Action and Custom.
  • Measurement group – contains functions used to perform test on the SUT, by measuring one or more values. The allowed step types for this group are Pass/Fail TestNumeric Limit TestString Value Test and Custom.
  • Sequences group – contains a list of sequences (in a MaTeLo sense, see above) which contains references to a set of functions defined in the two other groups.

These groups are presented in the Tree View of the Usage Model Editor:

 

Tree View with Test Bench Nodes.

There are two ways to define a Test Bench:

  • By importing a set of functions and sequence definitions from an existing test bench file of another MaTeLo project,
  • By creating manually each function and sequence definition through the Usage Model Editor GUI.

Importing a test bench definition file

Select File > Import > Import Test Bench Definition File.

Importing a Test Bench Definition File.

The Usage Model Editor launches the File Chooser that you have to use for selecting the Test Bench Definition File you want to import, as shown below.

The Test Bench Definition File is an XML file called TestBenchDefinition.xml. You can find it in the VersionX directory (X being the number of the last version of your usage model, refer to Usage Model Editor User Manual to get more information on the version mechanism) of the project from which you want to import the Test Bench definition.

 

Test Bench Definition File chooser.

 

You can also find some definition files examples in the MaTeLo installation directory. These files include definitions of TestStand built-in step types like Wait, Label, Statement, If/Then/Else, etc.


Once you have chosen the Test Bench Definition File to be imported, click on Open.

All the functions and sequences of the text bench will be imported into the current project.

If a function or sequence with the same name already exists in your Test Bench, a dialog box allows you to choose the expected behaviour of the import. You have the following choices:

  • Rename – the function will be imported with another name that you have to give through the following dialog box.
  • Replace – the existing function is overwritten.
  • Ignore – the function is not imported.

Already existing function dialog box.

New function name dialog box.

Once the functions and sequences are imported, the Tree View is updated and you will be able to associate each of them in your usage model.

Note that when saving your project, a new TestBenchDefinition.xml file will be created in your project directory.

Creating a new Setting or Measurement Function

Right click on one of the “Settings” and “Measurement” groups and choose New in the popup menu that is being displayed.

New function popup menu.

The Usage Model Editor launches the Setting or Measurement Function Definition window, which allows you to define all the characteristics of the function, as shown below.

  • (9) Module Function Name or Method Name– the name of the function, method, property or Sequence to be called in the code module (mandatory for LabWindows/CVI, C/C++ DLL, .NET, ActiveX or Sequence Adapter).
  • (10) Object Reference– in case of .NET Module or ActiveX Adapter, you need to define an object reference (mandatory for .NET Module Adapter or ActiveX Adapter).
  • (11) Create Option– in case of .NET Module or ActiveX Adapter, you need to define how to create the object (mandatory for .NET Module Adapter or ActiveX Adapter).
  • (12) Create from file– in case of ActiveX Adapter, if you have chosen the create option at "Create from file", you need to define the file.

Note: In case of a Custom Step Type, all the fields are enabled, but Function PathnameClass Name and Module Function Name are optional and have to be filled only if they are not already described in the Step Type.

You have then to define the function parameters as to be able to pass data to the step, using the (14) Parameter Management Table and the (13) Parameter Definition Table.

The Parameter Management Table allows you to add, remove and move up or down parameters. Once a parameter is selected, the Parameter Definition Table is updated and presents the properties of this parameter, as shown below.

  • (9) Module Function Name or Method Name– the name of the function, method, property or Sequence to be called in the code module (mandatory for LabWindows/CVI, C/C++ DLL, .NET, ActiveX or Sequence Adapter).
  • (10) Object Reference– in case of .NET Module or ActiveX Adapter, you need to define an object reference (mandatory for .NET Module Adapter or ActiveX Adapter).
  • (11) Create Option– in case of .NET Module or ActiveX Adapter, you need to define how to create the object (mandatory for .NET Module Adapter or ActiveX Adapter).
  • (12) Create from file– in case of ActiveX Adapter, if you have chosen the create option at "Create from file", you need to define the file.

Note: In case of a Custom Step Type, all the fields are enabled, but Function PathnameClass Name and Module Function Name are optional and have to be filled only if they are not already described in the Step Type.

You have then to define the function parameters as to be able to pass data to the step, using the (14) Parameter Management Table and the (13) Parameter Definition Table.

The Parameter Management Table allows you to add, remove and move up or down parameters. Once a parameter is selected, the Parameter Definition Table is updated and presents the properties of this parameter, as shown below.

  • (5) Name – the name of the parameter, as declared in the function prototype or in the subsequence.
  • (6) Type– the way data is passed on the function (see title [#_Parameters: “Parameters”] in chapter “TestStand concepts”). You have the choice between Function ParameterStep ParameterLocal Variable and Global Variable.
  • (7) Data Type– the parameter’s data type. The allowed types are (depending on the Module Adapter):
    • BooleanLongDoubleString or Object Reference for C/C++ DLL Adapter or .NET Adapter.
    • LongDoubleStringContainer or Object Reference for LabWindows/CVI Adapter.
    • LongDoubleStringContainer or Object Reference for LabVIEW Adapter.
    • BooleannumberString or Object Reference for Sequence Adapter.
  • (8) Pass Type – the way the data is passed on. You can choose between In and Out. If you choose In, data is passed from the step to the function. If you choose Out, data is passed from the function to the step (pointer); you have then to specify in “Default Value” a TestStand property in which data will be written (as Step.Result.PassFail for example).
  • (9) Default Value – the default value of the parameter.

When clicking on the “OK” button, if the dialog box is correctly filled in, the function is added to the “Setting” or “Measurement” group, the Tree View is updated and you can associate this function to the usage model.

                                                                                                          
Notes:

  1. If you use a TestStand Local or Global Variable to pass data, this variable will be created by MaTeLo in the sequence file. You will be able to use it later in other steps or code modules to recover or exchange data.
  2. If you use Step ParametersLocal or Global Variables, your code module must have a reference to the sequence context (refer to TestStand Manuals to be familiar with the functions or VIs prototypes to use with TestStand). You have then to declare a parameter with the Data Type “ObjectReference” and the Default Value “ThisContext”.
  3. LabWindows/CVI users: older versions of TestStand proposed a function prototype using 2 data structures (TestData and TestError). This prototype is obsolete, but you can still call functions using it. You will then declare 2 parameters with the Data Type “Container” and the Default Values “testData” or “testError”. However, in the produced step the check box “Pass Sequence Data” of TestData structure won’t be checked and you have to do it manually.

Creating a new Sequence

Right click on the “Sequences” group in the Tree View and choose New in the popup menu that is being displayed.

 

New sequence popup menu.

The Usage Model Editor shows up the Sequence Definition dialog box, which allows you to define all the characteristics of the sequence, as shown in below.

Sequence definition dialog box.   

To define a sequence, you have to do the following steps:

  • Fill in the Sequence Name (1) – the name of the sequence, as it will appear in the Tree View and in the association window (mandatory).
    • For each Step Group (3) (see title [#_Step_Group: “Step Group”] in chapter “TestStand concepts”), add functions references by double clicking on their name in the Available Functions List (2). You must add at least one function to the “Main” group, but “Setup” and “Cleanup” groups are optional.
    • The added functions appear in the Chosen Functions list (4) of each Step Group. You can reorder or remove them using the Up, Down or Remove buttons (5).

    When clicking on the “OK” button, if the dialog box is correctly filled, the sequence is added to the “Sequences” group, the Tree View is updated and you can associate this sequence to the usage model.

    Defining Test Bench Properties

    Right click on the “TestStand” node in the Tree View and choose Properties in the popup menu displayed.

         Test bench properties popup menu.         

The Usage Model Editor launches the Test Bench Properties window, which allows you to define the properties needed by TestStand to generate sequence files, as shown below.

Test bench properties dialog box.

Depending on your usage of the TestStand generator, you may have to change the following properties:

  • Template File Path (1) – to generate TestStand sequences, MaTeLo uses a sequence file from which it copies the Setup and Cleanup step groups, the local, the parameters and file global variables, the sequences (other that MainSequence). MaTeLo also creates each step (except for the custom step types) by cloning the steps from the template file. This template sequence file is delivered with the MaTeLo package, and by default the path is set to the MaTeLo install directory. If you want to customize the behaviour of a step, or add some functions to the setup or cleanup groups, or add some variables, you can:
    • Copy the template file to your project directory.
    • Modify this template file as you want. Be Careful! You mustn’t remove or rename the existing steps, or the generation may not work any more!
    • Change the Template File Path to this path by using the search button or by manually filling the text field.

If you use a TestStand version which is older than 4.2 (3.0, 3.1, 3.5 or 4.x), you also have to set the path to template files created for this format. They are also delivered with MaTeLo, and you only have to choose the right one with the search button.

  • Code Modules Root Directory (2) – the path to the directory that contains the code modules used by the test bench. If you set this path (by using the search button or filling manually the text field), you only have to set a relative path to your code modules when defining the functions. If not, you have to set absolute paths in function definitions.
  • Sub Sequences File Name (3) – If you define some sequences, MaTeLo creates it before generating the test case file in TestStand format. All sub sequences are created in the same sequence file, which name has to be set here. By default, the sub sequences file name is set to “SubSequences.seq”.

 

Associating Functions and Sequences to the Usage Model

                                                                                                        

Once the Test Bench functions and sequences are defined (see chapters Creating a new Setting or Measurement Function and Creating a new Sequence), you have to associate some of them to some transitions of the usage model. Each associated function or sequence of each visited transition during a test case produces a step in the generated sequence.

This chapter describes how to associate the functions and sequences to transitions using Usage Model Editor.

By “associating functions and sequences to transitions” we mean:

On a transition of the usage model, setting one or more references to TestStand functions or sequences in a particular order, and giving values to their parameters. These parameters can be valued by the content of an Input or an ERA of the transition, or by a constant value.

Adding Functions or Sequences references on a transition

Double click on the transition that you want to associate with the TestStand functions or sequences to open the transition window and click on the “TestStand Functions” panel, as shown below.

 

TestStand functions tab on transition dialog box.

Add functions or sequences by double clicking on their name in the Available Functions and Sequences List (2).

An added function appears in the Chosen Functions and Sequences list (3). You can reorder or remove a function using the Up, Down or Remove buttons (4).

An added Sequence also appears in the list, but it is followed by the list of each function that it contains, presented as a Tree. You can only reorder the sequence in the list, but not the functions in that sequence.

The title of the TestStand Functions Panel (1) indicates the number of associated functions.

When clicking on an added function (even a function within a sequence), its parameters appear in the Function Parameters Valuation table (5), as shown below.

Valuating the function parameters

Each Parameter has to be valuated, so that MaTeLo can set the value to the step function.

Parameter valuation table.

The parameters are presented line by line, in a 3-column table presenting:

  • The Parameter Name (2) – the name of the parameter as set in the function definition. This column is not editable
  • The Parameter Type (3) – the way to value the parameter. Assuming that one or more ERA(s) and an Input are created on the transition, you can choose to pass 3 types of value to the parameter:
    • Input – the value of the chosen input is passed to the parameter.
    • ERA – the value of the chosen ERA is passed to the parameter.
    • Value – the parameter is set to a fixed value
  • The Parameter Value (4) – Depending on the chosen parameter type, a text field or a combo box allows to set the value to pass to the parameter:
    • If the chosen parameter type is Input, a combo box presents a list of the input or sub-elements of the input (if the input is a sequence) that are compatible with the parameter data type. (Table 1 below presents the compatibilities between MaTeLo and TestStand data types)
    • If the chosen parameter type is ERA, a combo box presents a list of all the ERAs or sub-elements of the ERAs (if the ERA is a sequence) that are compatible with the parameter data type. (Table 1 below presents the compatibilities between MaTeLo and TestStand data types)
    • If the chosen parameter type is Value, a text field allows for filling the fixed value. If the parameter is a Boolean, a combo box allows choosing “true” or “false”.

Notes:

1- By Default, the parameter type is set to “Value” and the Value is set to the default Value set in the function definition.

2- The compatibilities between the parameter data types and the MaTeLo data types are the following:

 

 

Compatibilities between MaTeLo and TestStand data types.

For example: a TestStand function which has a String typed parameter can accept data from an Input or an Era of any type. A Boolean parameter can only be valuated by a Boolean Input or ERA.

3- If a parameter is valuated by an ERA which comparison type is complex (i.e. a comparison with two values, like GELE for example), you can choose to pass the low or high values of the ERA to the function. These values are presented under the form: “name_of_the_era.low” or “name_of_the_era.high”.

4- If a parameter is valuated by an ERA, the comparison type can be passed as a parameter to the function. If the parameter is “String data typed”, the comparison type is added to the Parameter Value combo box as “name_of_the_era.comp”.

Setting the Numeric Limits (Numeric Limit Test)

If you defined a measurement function out of which the Step Type is “Numeric Limit Test”, you have to set the ERA which specifies the one or two limit values the step uses to perform the limit comparison (see title “[#_Step_Types: Step Types]” in chapter “TestStand Concepts”).

Once you have selected a Numeric Limit Test Function in the “Chosen Functions and Sequences List”, an additional panel appears next to the parameter valuation one, called “Limits” (1). You only have to choose an ERA among the list appearing in the “Choose ERA combo box” (2), as shown below.

 

Comparison limits tab.

The “Choose ERA combo box” only contains ERAs (or sub-elements of a sequence typed ERA) that are compatible with a Numeric Limit Test: the MaTeLo data type of the ERA (or sub-element of the ERA) must be “Integer” or “Float”.

Setting the String Reference (String Value Test)

If you defined a measurement function which Step Type is “String Value Test”, you have to set the ERA which specifies the reference string that the step uses to perform the comparison (see title “[#_Step_Types: Step Types]” in chapter “TestStand Concepts”).

Once you have selected a String Value Test Function in the “Chosen Functions and Sequences List”, an additional panel appears next to the parameter valuation one, called “Limits” (1). You only have to choose an ERA among the list appearing in the “Choose ERA combo box” (2), as shown below.

String reference tab.

The “Choose ERA combo box” only contains ERAs (or sub-elements of sequence typed ERAs) that are compatible with a Numeric Limit Test: the MaTeLo data type of the ERA (or sub-element of the ERA) must be “String”.

                                                       

Generating TestStand Sequences

 

Once you have modelled the system to test, Testor will randomly generate some test suites. Those test suites then need to get ran by an operator or an automatic test bench as TestStand.

This chapter describes how to save test suites as TestStand sequence files (.seq) using Testor.

Saving Test Suites as TestStand Sequence Files

This chapter assumes that you are familiar with test suite generation. If not, please refer to Testor User Manual.

Launch Testor, open a project and generate one or more test suites. When the generation is done, the “Generation Report” window is launched, as shown below.

 

 

Generation report window.

Click on “TestStand Save” button to save the test suites as TestStand sequences.

A file chooser is displayed and allow to choosing the sequence file name and the directory in which the sequence files will be saved, as shown below. By default, the selected directory is the “TestSuites” folder in the current project.

TestStand sequence file chooser.

Each test case of a test suite will produce a sequence file. Note that the sequence files will be named with the chosen name, suffixed by an underscore (“_”) and the number of the test case. For example, if you generate a test suite of 5 test cases, and that you choose to name your sequence file “mySequence.seq”, MaTeLo will generate 5 files, named “mySequence_1.seq” to “mySequence_5.seq”.

If the TestStand generation was successful, a popup message informs you, as shown below.

 

Successful saving popup message.

Otherwise, a message popup informs you that an error occurred, and a message explaining the error is displayed into the “Messages Zone” of the main screen of Testor. You have to correct your model or modify your code modules so as to be able to generate correctly.

Once you have successfully generated TestStand sequences, you can run them on your Test Bench…