The glue for an automotive test automation ecosystem

ETAS, a member of the Bosch group, is developing an open-source-based test automation framework for automotive systems together with renowned partners such as Volkswagen/CARIAD, Mercedes Benz Tech Innovation, and AVL. The project is being realized under the umbrella of the Eclipse Software-defined Vehicle Working Group and named “Eclipse openDuT” (DuT=Device under Test). The aim is to create an lock-in free validation & verification (V&V) solution that enables automated, location-independent, flexible automotive testing without great expenditure of time and money.

Today, the tool landscape in the automotive testing sector is quite fragmented and predominantly offers proprietary solutions that are incompatible with one another. Users such as automotive manufacturers and suppliers quickly become dependent on individual providers (vendor lock-in effect). At the same time, many commercial testing tools already consist largely of open-source components – a hybrid situation that causes dissatisfaction among users.

The Eclipse openDuT framework aims to automate and integrate the testing and validation process for automotive software and components within existing infrastructures. This framework will serve as the foundational basis for testing automotive systems, supporting various test applications, including security, safety, and functional tests for individual components and networked systems, as well as for homologation purposes (e.g., type approval in accordance with UN-R155).

The involvement of experienced automotive industry experts ensures practical relevance and domain knowledge, including the integration of automotive specificities (e.g., DuT management or rest bus simulations) and a focus on extensibility for integration into native systems (e.g. OpenTestSpecification [1] or AUTOSAR). As a result, users within the test automation framework can continue to utilize proprietary test applications available on the market and define and integrate their individual test methods, such as black or grey box tests, across geographically dispersed test benches.

The separation concept of differentiation and non-differentiating software can be explained by Fig. 1. With fulfilling predominantly the test facilitation aspects, Eclipse openDuT represents the non-differentiating part of the V&V framework. The benefit of an open-source based testing infrastructure lies in the opportunity to include test applications of various context within the test execution layer, i.e., integrating market available test tools (commercial-off-the-shelf) with different license types (open source or proprietary) as well as test suites that are custom and engineered for a very specific validation purpose (e.g. a secure boot function). In addition, the test facilitation and execution should be accompanied by an integration into the user’s test infrastructure, e.g., digital workbenches or test workflows, and an appreciation of the business context, e.g., by test specifications based on system requirements or qualification of the framework for productive usage or type approval audits. The DuT layer together with the test execution and service & integration layer form the differentiating part of a test automation framework. At this point the specific validation requirements of the customer, e.g. specific cybersecurity goals of a component, are met by the standardized test infrastructure.

As shown in Fig. 2, the Eclipse openDuT framework consists of an entire IP-routed network infrastructure that is centrally managed and interconnects several edge devices (peers) by a secure Virtual Private Network (VPN). The edge devices can connect geographically distributed devices under test (DuT) with each other and thereby allow direct ECU-to-ECU communication through the respective automotive protocols e.g., CAN or SOME/IP. Furthermore, the peers can measure and control each connected DuT by automotive-specific means, e.g., rest bus simulation based on AUTOSAR configuration files (ARXML), control of an external power-supply, initiate a flashing procedure or trigger the clamps 15/30 via serial communication. In this way, users have maximum flexibility and freedom to adapt the framework to their individual use cases, to integrate it into automated workflows, or even digital automotive workbenches. Eclipse openDuT can integrate physical ECUs as single devices or clusters within a local testing environment, per cloud, on premise, or in a hybrid deployment. In future, it is planned to implement an interface for Software-in-the-Loop (SiL) environments to run the tests also on virtual ECUs. A setup, as shown in Fig. 2, enables the chaining of different cybersecurity validation steps with different test applications, e.g., a proprietary cybersecurity protocol fuzz tester together with a customized functional test suite written in Python, to cover a significant proportion of type-approval-relevant tests for vehicular components. To sum up the main advantage of Eclipse openDuT, the framework allows to combine individual tests from different origins in a single open framework without limitations to ensure an automated and integrated tool chain.

The project was first presented at the EclipseCon conference in Ludwigsburg, Germany, on October 16, 2023. Following official acceptance by the Eclipse Foundation, the implementation of the open-source-based test automation framework is now progressing apace, with ETAS and other partners already providing code (GitHub repository). In view of the cybersecurity and functional safety requirements of future software-defined vehicles, automotive testing must become faster, more efficient, and easier to integrate into existing landscapes, and this initiative creates the prerequisites for this.