Thursday June 29th 2023
Why is this calculation wrong again? Why is the application crashing again when this message is sent? Why is my app hanging, again? Who doesn't recognize these complaints, as a user or tester? Is it really so difficult to prevent these errors?
The solution, supported by theory and practice, is obvious: more and better testing. Couldn't the error you ran into have been found and prevented earlier in the process?
Why don’t we test with more test-cases?
What stops organizations from testing more? Often people are not really aware that many more test cases are needed. So, in this case, it is not being managed either. This is necessary, especially given the increasing complexity of software. Time pressure at the end of the process and limited resources also play a role.
Is software really that complex? Software complexity is invisible and difficult to grasp. Let’s use a map as a metaphor for software: imagine the software as a map with many highways, B-roads, country roads, bicycle paths, tunnels, bridges and ferries. A somewhat interesting piece of complex software is certainly as complex as the road network of the Netherlands. In practice, testing is often done on a portion of the main functionality, say a dozen or so highway routes. This is also known as risk-based testing. With any luck, some exceptions like bridges and ferries are also tested. Testing is mostly done manually, and with each new version of the map (the software), one jumps in the car and drives down the same collection of routes.
If the complexity of your software is comparable to the Dutch road network, and you want to be sure that the map matches reality, you can imagine that a dozen, a hundred or even a thousand tests are not enough. It’s not so surprising that something goes wrong from time to time.
How can you do better?
To test more and better, test automation is necessary. This has been known for a long time. Back in 1999, Kent Beck wrote his famous book: eXtreme Programming Explained, in which he describes best-practices from the decades before, such as test automation (in particular unit-testing). Only manual testing is really not enough anymore, it takes too much time as well.
Good techniques that can help with this are Behavior Driven Design (BDD) and Test Driven Design (TDD). TDD means that you start with a test case before you write the code. This ensures that for every line of code, there are test cases that describe and cover the functionality. A special form of this is BDD, which focuses on the behavior scenarios from a user perspective. These are also defined before the programming starts. During construction, these BDDs are maintained.
BDD and TDD both ensure:
- Much more testing is done, not just a few highways and exceptions, but most (if not all) of the roads built.
- (Automatically) testable systems are created.
- More automated testing. Not just manual testing by jumping in the car, but working with robots and drones.
BDD and TDD are tools, the goal is to arrive at a sufficiently thorough approach to say something meaningful about software validation and verification.
- Validation: (automated) demonstrating that the software does the right thing, what the domain expert needs to automate processes (did we build the right thing?),
- Verification: (automated) demonstrating that the techs built the software correctly, without bugs and crashes (did we build the thing right).
The Axini approach
BDD and TDD (scripting in general) are currently the most widely used techniques to automate testing. The challenge these techniques add is the many test cases that need to be maintained efficiently. This is something Axini can help with very well. Axini implements the test-first approach by creating behavioral models, also known as Model Driven Design (MDD) with Model Based Testing (MBT), the next step of BDD/TDD, but that's something for another article.
Want to know more? Contact us or follow us on LinkedIn.