The MBT (Model-Based Testing) is a test approach associated with test engineering process based on Model Driven Engineering.
Test phases are the last stages of system development. They are therefore strongly constrained by delivery periods. In general, they are subject to all of the delays accumulated during the overall project. Currently, test phases still correspond to 30% or even to 50% of the total cost of system development and thus represent an important point for possible improvements.
Currently, the verification & validation process having no systematic or automatic process is often underestimated and outsourced to reduce costs. All these factors affect the quality of current products.
Testing is a difficult activity. Indeed, it is not possible to test everything because the quantity of test cases to be applied seems infinite for the majority of modern systems. The difficulty of testing lies in the selection of relevant test cases for each phase of validation. It also lies in the absence of a true reflection of test optimization guided by the maintenance of the final quality while minimizing the costs.
In this document, we will present a technology which is more and more used in industry to avoid bugs, to improve quality and reduce costs: Model-Based Testing.
What is Model-Based Testing?
The software industry has recently lived through a methodological revolution by integrating modeling as an engineering and development tool. The models are used in the phases of design (approach indicated by the term Model-Based Design) to allow quasi-systematic methods of generating code starting from models, with tools like Simulink® in the automotive industry and Scade® in aeronautics or the railway industry.
Model-Based Testing is a systematic method to generate test cases from models based on system requirements.
First, what is Testing?
From ISTQB - International Software Testing Qualification Board, Testing is the process consisting of all life cycle, both static and dynamic, activities concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.
The goal of testing is failure detection: the differences between the behaviors of implementation and what is expected on the basis of the specifications. There are different levels of testing:
The unit testing is tests performed by the developers themselves. The goal is to validate code quality and performance.
The integration test is testing performed to expose defects in the interfaces and in the interaction between integrated components or systems.
The validation test is tests performed on the entire system.
To do testing, we can use the Model-Driven Engineering (MDE), a software development methodology which focuses on creating and exploiting domain models. Model-Based Testing is founded on system modeling and is part of MDE.
Model-Based Testing was mainly deployed in the industry on the descending phase of the V life cycle, i.e. from functional specifications to implementation through design. Having proved the benefits of efficiency and profitability, Model-Based Testing is now used on the ascending phase of the V life cycle, as verification and validation steps.
Product Life Cycle
From ISTQB - Validation is confirmation by examination and through provision of objective evidence that the requirements for specific intended use or application have been fulfilled.
From ISTQB - Verification is confirmation by examination and through the provision of objective evidence that specified requirements have been fulfilled.
As testing method, there are black-box testing (functional test), white-box testing (structural test) and grey-box testing (mixed strategy between black and white-box testing).
Model-Based Testing is a black-box approach, i.e. this is a method that examines the application functionality without peering into its internal structures or tasks.
According to Wikipedia, Model-Based Testing is application for designing and optionally also executing artifacts to perform software testing.
Models can be used to represent the desired behavior of a System Under Test (SUT), or to represent testing strategies and the SUT environment :
Test Generation Tool
System Under Test
Main Model-Based Testing steps
Model-Based Testing is a systematic method to generate test cases from models based on system requirements. It allows you to verify that your system has been designed entirely according to its functional requirements (the outputs of the SUT are the same as the predicted by the tests).
The testing effort is made by test engineers during the design of the usage model.
The MBT approach allows doing a SUT tests representative number defined by the user. The verification is more complete in comparison with the traditional testing, because the user can assemble several requirements, use equivalence classes, use statistics and generate a large number of scenarios, which will cover a broad test spectrum that tests writing manually.
If we assume that the program verifier requires a significant amount of input to achieve good results (for example, very precise specifications of the system and each module within the system), then the cost of full verification may not be cost-effective for some non-critical systems. In this case, MBT may be a viable alternative.
The effectiveness of Model-Based Testing is highly increased with automation that is to say that the capability to automatically translate test cases into executable test material. But we will discuss about that later in this document.
The Model-Based Testing involves:
Creating a model based on SUT functional requirements
Automatically generate test scripts and test cases
And then, with the right test environment, execute the test cases automatically and generate analysis & reports.
Model-Based Testing Concept
A usage model of the SUT is designed based on the system requirements or existing specification documents. The abstract model represents the intended behavior of the system under test (SUT).
Then, to generate test cases, the user is choosing his test strategy depending on the maturity level of the project. The tester can focus on the most used features and most probable SUT’s behaviors, or focus on the most SUT critical and complex parts or profiles, or generate the test cases that cover all the requirements with the lowest test effort, or use a random testing technique.
The tester can also test a part of the system with the specific specifications and ignore all the rest of the SUT. When the project is more mature, he completes the SUT model and start again with a more complete test strategy and therefore more test cases.
Then, test scripts (test procedure specification) are executed based on the test suite (set of several test cases) generated to be interpretable by the test environment. The test scripts are ran on the SUT and finally, a series of metrics like quality, test coverage, reliability are offered at the user.
Based on the results, i.e. a comparison between the SUT behavior output and the result of the executed test cases, the tester team is able or not to modify requirements to solve failures and to improve the system.
Thanks to the Model-based Testing approach, the user can design a model, define test selection criteria, choose a test strategy, generate and execute test cases on the SUT.
If the user doesn’t desire automated test, he is able to do manual testing from test cases. But, the ultimate step is to run the test scripts automatically thanks to the appropriate test environment.
Why using Model-Based Testing?
Model-Based Testing allows a better understanding of the application and enables more agile testing.
Depending on the test strategy, the user can generate a very large account of tests.
Easy test case maintenance
As much test cases as wanted
No programming skills needed
Early bug detection
Increased bug count
Time to address bigger test issues
Improved tester job satisfaction
How to do a Model-Based Testing application successful?
When a company makes Model-Based Testingchoice, it is a fundamental decision and there are some prerequisite to be successful and optimize the choice.
For a good modeling, the team has:
To be aware or to have a knowledge about the SUT
To have basics knowledge in testing and Model-Based Testing tool philosophy
To have SUT functional requirements
To have a good level of abstraction and reflexion
To know the Model-Based Testing tool used by his company.
A question that the testing team can ask is: do we have to model from the requirements or from the test bench?
Answer: first, the test model should be made based on the requirements for the basis architecture of the system and second on the test bench to optimize the model as much as possible.
From ISTQB - Test automation is the use of software to perform or support test activities, e. g. test management, test design, test execution and result checking” !
Automation Test Framework
Test Cases from the MBT tool
System Under Test
Model-Based Testing and Test Automation
There are a numerous industrial test frameworks on the market place like CANoe, NI TestStand, NI VeriStand, ProveTech, EXAM, eggPlant, Selenium, webdriver, Scade, Matlab, QTP/UFT, or languages (Python, C++, C#, VB...).
Test execution platform is the final step when you are using Model-Based Testing technology. It can execute some repetitive but necessary tasks in a formalized testing process already in place, or add additional testing that would be difficult to perform manually (time consuming). Testers can run the test scripts on the appropriate test environment, quickly and repeatedly. It offers a real opportunity to pass a large number of test cases and to increase the quality, the time and the test coverage.
The functional test automation and execution are now more feasible because of technological, organizational and budget maturity. Indeed, industrial or IT companies know now that test frameworks are reliable, the methodology & knowledge are well-known, and the cost is reasonable in comparison with the quality and agility that we obtain.
Model-Based Testing today?
More and more test conferences organized around the world have Model-Based Testing topics on their agenda, like: Model-Based Testing and Agile, Model-Based Testing and the Cloud, Model-Based Testing for Dummies, Model-Based Testing and the Test Ecosystem, Model-Based Testing Challenges…
We also can see several companies (industrial and IT) choosing this technology, but changing the mentality are not easy and is time-consuming. Testers have to be included at the high level of product life cycle. Then, failures can be find very early, requirements and specifications can be changed and the cost & quality product will be better. Modeling requests time and training.
Since some years, modeling is more used, particularly in domains like automotive or telecom for example, because quality and cost are crucial and mentality has changed.
Model-Based Testing approach has modified the testing process. The process flow is more structured and allows detection of specifications errors in its early stages. Thus, it allows the identification of new test cases by requiring an increased exchange with the system specification team and allows the automatic generation of test scripts. The test designer is able to generate as many test cases as necessary to ensure an enhanced coverage and reliability of the tested functionality.
Early validation of requirements is a known “best practice”. The fundamentals of Model-Based Testing are now well-known on the industrial/IT domains and on the research part too. Everybody agree to detect bugs as soon as possible in the product life cycle. Since several years, there are software tools dedicated to MBT proposed to dedicated people.
Minds are changing and Testers have the capability to “jump the gap” to choose Model-Based Testing adoption with Matelo!
Development and testing are from the same sources in which the need is defined by functional requirements. In the MDE approach, the functional requirements are implemented in a design model, aiming to find a solution for the formalized need. In the MaTeLo MBT approach, these functional requirements are expressed in a usage model: the purpose is to validate that the solution sufficiently meets the formalized need. The functional requirements are therefore optimally employed thanks to the use of models, but with different points of view (implementation, writing scenarios) and different objectives (solution, validation) for finally evaluating an achievable solution using test scripts carried out in an automatic test environment.
The MaTeLo model is used to represent the desired behavior of a System Under Test (SUT), together with the testing strategies and the test environment.