Friday February 2nd 2024
Every year, NWO and Lorenz Center organize the ICT with Industry workshop. This is a 5-day workshop focused on devising innovative solutions for scientifically challenging, commercial business cases. At Axini we are always looking to innovate our model based testing platform. ICT with Industry gives us the opportunity to engage with smart researchers.
As an industry partner we provide cases for a team of smart researchers to work on to produce a solution. This year we had the luxury that 2 of our cases were accepted: one case about test-automation for 3D games and one about the application of GenAI and/or Large Language Models (LLMs) in model based testing. An important criterion for Axini is that at the end of the week we want to see a working prototype.
Both teams worked extremely hard and achieved remarkably interesting results, including working prototypes.
GenAI/LLMs and model based testing
The team that worked on GenAI for MBT
As you can imagine, by now everyone and their uncle is interested in GenAI, including us. In particular, we are interested in the use of GenAI for the creation of Axini models. The team investigated this and produced an interesting and viable approach to use Large Language Models (LLMs) to convert requirements and specification documentation (written in natural language) into Axini models. They also demonstrated that this approach works for converting source code into Axini models. As such, the team developed an LLM pipeline that converts the natural language documents into Axini models by converting the documents into source code as an intermediate step. Although the models are not always 100% correct, they are a great starting point. It is easy to extend and enhance (parts of) the models delivered by the AI approach.
Model based testing of 3D racing games
The team that worked on MBT of 3d games
The systems that we model and test generally have clear interfaces. When it comes to 3D games this is not the case. There is a lot of implicit knowledge, that is easy to understand by humans, but is difficult for computers. For example, with racing games, the controls are easy: accelerate, break, left, right. However, if you let a computer drive around a circuit with these instructions, the car is more off the track than on it. Even if one finds bugs there, they are of lesser interest to game creators. So, we were looking for a way to describe to our platform what the intention of the game is.
The team that investigated this case chose an open-source 3D racing game as the system under test. They produced the idea to introduce a game task-manager with higher level control of the game that our platform can tap into. As a result, our platform can drive the racing car from point to point on a game map, while checking that the game state is coherent, i.e., it does not freeze or lock-up and collisions are properly detected.