It’s common to view domain testing as a mechanical test technique to create tests. It can be just that…
but it can also be more…
What is it
The domain test technique is one of the most common used test techniques and can be a very powerful technique used to evaluate a product. Domain testing is a combination of test design techniques that focus on any data that is used by the software. Both data that is used as input to the software as well as the output from the software is interesting to this technique. By identifying variables, analysing them and designing tests based on equivalence classes, value selection and combinations can we perform domain testing.
The main advantage of the technique relies on the impossibility of complete test coverage. Imagine a calculator. Making every possible calculation possible would take forever. Domain testing lets us group values together that ought to behave the same way and only test a few of them. So, domain testing is a tool that enables us to choose best representative values to test and reduce the number of tests.
Variables
A variable contains a value or values that is liable to change. Different types of variables exist that will affect the software in different ways.
External variables like operative systems, web browsers and screen resolution are variables that the software relies on to behave as expected.
Internal variables are product specific and reside only within the product and/or other integrated systems. These can be both visual and hidden.
Visual variables can be seen and can be modifiable to the user. Commonly they are presented in the user interface but can also be in a configuration file.
Hidden variables are those that we don’t see through a user interface. They are in the code and not displayed to the user. They serve “under the hood” to support the logic of the software. The hidden variables can be of interest since they may change which code that is executed and involved in algorithmic logic that in turn affect the quality of the software.
The variables of interest are those that affect how the software under test is behaving. Internal, external, visual or hidden it doesn’t matter. If they affect the software and its quality, they are of interest. The time you have to test is limited and you have to prioritize which variables to test and how deep test coverage you want for each variable.
Variables and software
Photo by True Agency on UnsplashDifferent variable types and its values are handled in different ways by the software.
External variables are rarely in scope of the testing project but how they work together with the product usually is. External variables create the foundation of which the software shall run. Different end users might have different configurations which makes these variables liable to change. One of the most common purposes of domain testing external variables is to evaluate compatibility with the product.
Valid values of the internal variables are probably the most common in domain testing. However, depending on the context both invalid and valid values are of interest.
Invalid values of the variables need to be handled by the software. Most software has filters implemented to achieve this. Filters are logic in the software that checks the input for invalid values. For example, if a variable that is supposed to contain the value of age contains letters, the filters should detect and handle it. The filters are commonly implemented using regular expressions, but not always. The filters can be implemented in various places in the code. Common filters are input filters, API filters and function filters. More on this topic in part 2.
When to use it
From experience I have used the domain test technique in all projects I have been involved in. Only the level of formality in the design has varied
How-to
Photo by Helloquence on UnsplashPerforming domain testing can, as any test technique, be done in a formal or informal way. The formal design and models are described below. You design by modelling and documenting the tests before or in parallel to executing them.
The informal way is more on the fly and the design is created in your mind. The design doesn’t have to be different but, in my experience, the formal approach usually gets more details and depth in its testing. The formal approach takes more time.
No approach is right or wrong. You need to decide if the formal approach is worth its time. Consider risk, test coverage and the value that the formal approach might give in relation to time and other test activities.For me, domain testing can be divided into different activities:
- Identify variables
- Identify equivalence classes
- Decide coverage strategy
- Design tests
- Execute tests
Identifying variables
In order to evaluate what variables to test and in which way we first need to be aware of them. Variables can be found by touring a user interface, specifications, code etc. Variables that is used as input are usually the most common, but output variables are also of interest. However we try to identify them, we can be pretty sure that we have not found them all. Focus on finding the variables that matters to your context.
Touring the user interface is a common technique to identify variables. The tester navigates through the user interface with the purpose of identifying variables and to learn how they can be modified and what they affect. Personally, this is my most common way of identifying variables.
Touring specifications in order to identify variables can be a good idea especially when the specifications include details about possible values and requirements connected to that variable.
Touring code is a powerful approach to identify hidden variables. Personally, I find that combining user interface tour with code touring is a powerful approach as you get insights to what is happening in the code when actions are taken in the user interface. Typically, there is a lot more going on than the user interface implies.
Controllability
When a variable has been identified we need to understand how we can modify the value of that variable. Sometimes we can modify its value at different places in the user interface or configuration file etc. If you don’t know if the different places where you can alter the variables value, use the same code. There might be a good idea to do separate domain tests for each place. The design of the tests might be the same but the ones you choose to execute might be different. For example, if you do deep testing at the first place you might be content with a lesser coverage at the second place. The tests and coverage you choose for the second place should depend on your risk assessment.
Observation
Photo by Mick Haupt on UnsplashAfter tests has been designed and executed, we need to evaluate the results. A certain variable value effect on the software might not always be easy to observe. We need to understand what and where affects will appear. A certain variable value effect can be visible immediately after the tests has been executed but also shown later in the software states.
A risk-based observation approach lets you imagine risks related to your test and look for those problems.
Personally, the user interface, storage entity’s, and logs are common places to observe and evaluate a test of a product.
The user interface often shows the variables effect and output. Try to identify where the variables value will affect behaviour or output. The affect can consist of output variable values, graphics rendering or pretty much anything. Try to keep an open mind when observing so you don’t miss unexpected behaviour
Databases, files and other data storage can be a great place to observe output result details. Especially when there are a lot of values that are getting created, updated or deleted.
When encountering complex software implementations where the variable plays a role there might be a good idea to ask the developers to log software actions. For example, can you log function usage, variable values etc.
Software testing oracles
We can use software testing oracles to identify possible problems. See the link below. But even when we are unsure about what might be a problem or not, the things we learn about how the system behave is information that is valuable. For example, if something is working well or in a certain way, that is also valuable information to project stakeholders. The things we learn can also be an inspiration to generate new test ideas or to raise questions about it.
Stay tuned for part 2: We will be looking at single variable equivalence classes, value selection, coverage strategy and design
References and further reading
https://en.wikipedia.org/wiki/Regular_expression
https://www.developsense.com/blog/2012/07/few-hiccupps/
http://www.testingeducation.org/BBST/testdesign/