Inside the Interoperability Test Bed with Miguel Diez Blanco and Costas Simatos

Interoperability is often presented as a matter of policy, standards and architecture. But once common specifications are agreed, a more concrete question follows: can the systems built by different administrations, countries or organisations actually work together? 

The Interoperability Test Bed was created to answer that question. Today, according to DIGIT figures shared in this interview, around 70 known projects and user communities use it, while its open source components have been downloaded more than 4 million times.  

To understand what the Test Bed does and where it is heading, we spoke with Costas Simatos, Technical Lead for the Interoperability Test Bed, and Miguel Diez Blanco, Team Lead for the Interoperability Enablers and Open Source at DIGIT’s Interoperability and Digital Government Unit. Costas has worked on the solution since 2015, while Miguel brings the wider DIGIT perspective on how reusable services can support interoperability across Europe. 

Q: For readers who are hearing about the Interoperability Test Bed for the first time, what is it? 

Costas Simatos: The Test Bed is a set of reusable solutions that help projects check whether IT systems conform to agreed technical and semantic specifications. It focuses specifically on how IT systems interact with each other: the data they use, the interfaces they expose and the processes they need to follow when communicating with other systems. 

Under the Test Bed umbrella, we provide two main types of solutions. The first is a set of validators: reusable software that can be used to create simple web applications for data validation. The second, and in many ways the flagship solution, is the full conformance testing platform. This is a web application that can be used to organise, execute and monitor tests designed to put IT systems through realistic business processes. 

In both cases, the purpose is the same: to help project teams verify that what they have built can work as expected in a wider digital ecosystem, before it becomes part of a service that other administrations, organisations or users depend on. 

Q: What problem was the Test Bed originally created to solve? 

Miguel Diez Blanco: The idea goes back to the ISA programme, when a lot of work was being done to promote interoperability across the EU. One of the key pillars was the creation of specifications that would allow IT systems to enable cross-border digital public services. These specifications are, in a sense, contracts between systems: they define what each system needs to do so that services can work together.  

What was missing was a clear and reusable way to test whether those contracts had actually been implemented correctly. Different administrations and project teams could read the same specification and build their own implementation, but there still had to be a way to check whether those implementations were correct and whether the systems could interact as expected. 

At the very beginning, the Test Bed was conceived almost as an easy-to-access pool of resources: infrastructure and hardware that project teams could use to test their systems. Quite quickly, however, it evolved into something more useful: a set of ready-made solutions that projects could reuse for validation and conformance testing. It filled the gap of interoperability quality assurance. 

That was important because many projects were facing the same need and were starting to build their own testing services from scratch. The Test Bed was created to avoid that reinvention. Instead of each project building its own tool, it provides reusable solutions that can be adapted to different contexts. 

For a non-technical audience, the contract metaphor is very helpful. In interoperability, machines also communicate through contracts, or specifications. Testing whether these contracts are correctly implemented can be time-consuming, and what started as an isolated need became more and more relevant as interoperability projects multiplied. 

Q: Miguel, how would you explain conformance testing to someone who is not technical? 

Miguel Diez Blanco: A simple analogy is sending a letter. You have a letter, you put it in an envelope, you write the recipient, you include the sender, you add the stamp and you make sure the package can actually be sent. Before the letter moves through the system, someone checks whether the necessary elements are there. 

Conformance testing does something similar for IT systems. It checks whether the data, interfaces, message exchanges and processes are all following the agreed rules. Are the right elements present? Is the recipient correctly identified? Is the message being sent in the expected way? Does the system respond as the specification says it should? 

If everything is in order, the system can participate with greater confidence in a wider digital service. In the postal example, the letter can be sent. In a digital public service, it could mean that one national system can safely join a cross-European exchange with other systems. If something is not in order, the Test Bed helps identify what needs to be corrected before the system goes further. 

Q: Can you give a real example of this in action? 

Costas Simatos: One example is the Once-Only Technical System, where Member States need to exchange evidence and certificates across borders. There are detailed requirements describing what messages look like, how they are sent, how systems respond and which sequences need to be followed. 

The Test Bed is used there for conformance testing, both in connectathon-style events and in standalone testing. In a standalone test, a system does not necessarily need to test with another real national system. The Test Bed can act as the counterpart. It can receive requests, send requests, check the data, verify the sequence of messages and make sure the different aspects needed to participate in the system are correctly implemented. 

Q: What is the difference between data validation and conformance testing? 

Costas Simatos: Data validation is the simpler concept. It checks whether a specific piece of data follows the expected structure, syntax and content rules. An electronic invoice is a good example. One of the oldest and most popular validators we have online is an electronic invoice validator, and the idea is straightforward: the invoice is expected to be structured in a specific way, using a specific syntax and specific rules. 

A validator can check whether the data is actually XML, whether the required elements are present, whether the seller and buyer are defined correctly, whether the items are listed as expected and whether totals match the individual amounts. It is essentially checking that the data itself is correct according to the rules. 

Conformance testing is broader. It includes data validation, but it also checks everything around the data: the interfaces through which the data is sent and received, the sequence of messages, the business process being followed and the consistency of the whole interaction. It can verify not only that a valid invoice exists, but also that a system can generate the right invoice, for the right context, at the right moment in a test. 

That context matters. A test might require an invoice for a specific product, a specific recipient or a date after the start of the test session. The goal is to make sure that a system can produce valid data on demand, rather than simply reusing the same valid example again and again. Data validation is often a building block inside conformance testing, but conformance testing asks whether the whole system behaves correctly. 

Q: What kinds of projects are the best fit for the Test Bed? 

Costas Simatos: Any project that involves multiple IT systems exchanging data based on common requirements can benefit from the Test Bed. If a project defines data, it can benefit from a validator. If it defines how that data should be exchanged, it can benefit from a conformance testing service. 

The Test Bed is especially useful when there is a project owner, lead or facilitator who needs a helicopter view of implementation. It provides reporting and monitoring: who has tested, who has not, which tests have been completed, how far each participant has progressed, which version of a specification has been used and whether a participant supports all expected messages or only a subset. 

That makes it useful not only for developers, but also for the people responsible for steering a digital service community. It gives them visibility over the testing process and helps them understand whether implementation is progressing as expected. 

Q: Why is this particularly relevant in Europe? 

Miguel Diez Blanco: The more Europe integrates, the more it needs to interoperate. Europe has been built differently from other regions of the world. We have different administrations, different systems, different legal and organisational contexts and different implementation choices. In that sense, Europe is naturally required to interoperate. 

That is why the Test Bed will continue to play a key role. The more we develop cross-border digital public services, the more important it becomes to make sure that the systems behind them are not only connected in theory, but able to work together in practice. 

Q: Which success stories show the value of the Test Bed? 

Costas Simatos: A recent example is the work around European Digital Identity Wallets. The team needed to prepare a testing event in a limited amount of time and used the Test Bed to power that event. They were able to create a comprehensive testing service, with tasks and scenarios, and use it for the last EUDI Wallets Launchpad event. The feedback from participants was very positive. 

What makes this example interesting is what happened afterwards. Based on that experience, the team started building a more permanent conformance testing service using the Test Bed, so that wallet implementers would have a service they could connect to and test against on an ongoing basis. For me, that confirmed that the Test Bed can be picked up by a project team, adapted to a demanding context and used to deliver value quickly, with limited support from our side. 

Miguel Diez Blanco: Another strong example is the World Health Organization (WHO). The Test Bed was used in relation to vaccination certificates for the Hajj pilgrimage, and later in a larger testing context in Latin America linked to yellow fever vaccination certificates. What I like about this example is that it shows interoperability as something that can be linked directly to public health, not only to efficiency or cost savings. It is about making sure certificates issued by different countries contain the right information and can be recognised across borders. 

Q: How widely is the Test Bed being used today? 

Costas Simatos: We are aware of around 70 projects or user communities using the Test Bed based on their direct contact with us, although the real number is likely higher because the software is freely available and open source. Many users download it, follow the documentation and deploy it without contacting us, so we only know part of the picture. 

The usage figures also show the scale. The software components have been downloaded more than 4 million times from the Docker Hub. The hosted validators operated by DIGIT have processed around 5 million validations, and the conformance testing platform has supported more than 100,000 tests. 

Q: The Test Bed has a very active release rhythm. What drives that? 

Costas Simatos: We aim for a feature release roughly every three to four months, with more frequent releases when needed. The main driver is user feedback. The Test Bed is used by many different projects, with very different needs and setups, so new requirements constantly emerge. 

Sometimes users want to customise how people access the platform. Sometimes they need a feature to align better with their own process. Sometimes they prepare a milestone and need a stable release at a specific moment. We try to prioritise this feedback and, where possible, align our release schedule with the release schedules of the communities using the Test Bed. 

There is also the experience we have built on our side. Because we see the Test Bed being used in many different contexts, we start to identify recurring needs: things that are missing, things that would make the platform easier to use, or improvements that users may not have asked for directly but that we know would bring value. That combination of user feedback and product experience is what keeps the release rhythm active. 

Q: Where is the Test Bed heading next? 

Costas Simatos: One major direction is better support for peer-to-peer testing events, where multiple teams of developers come together and test live with each other. The Test Bed is already used in this context, but we want to make this kind of testing a first-class feature: easier to organise, easier to monitor and easier for participants to follow. 

Another direction is AI. The Test Bed has many capabilities, but there is naturally a learning curve when people want to use the full range of what it can do. In the future, we want to make it easier to integrate the Test Bed into AI-based workflows. For example, an AI agent could interact with the Test Bed, help set up testing scenarios, prepare tests or guide users through configuration steps.  

Q: What about the Interoperable Europe label? 

Miguel Diez Blanco: The Test Bed has been proposed for the Interoperable Europe label and has strong support for it.  

For me, the Test Bed is a natural candidate for the label for two reasons. 

First, it was built to be reused. It is not a solution that happens to be reusable after the fact; reuse was part of its purpose from the beginning. Second, it fills a niche that many IT projects need but do not always place at the centre of their policy or legislative work: conformance testing and data validation. 

Almost every project involving IT systems includes data exchange between systems. In those cases, testing whether the exchange works correctly is essential, but it is rarely the main purpose of the project. That is precisely where a tried-and-tested, flexible solution such as the Test Bed can help. There is a use case for it in a very wide range of IT system developments. 

Q: If readers remember only one thing about the Test Bed, what should it be? 

Costas Simatos: If your project exchanges data, or if your systems need to interact with other systems, do not neglect conformance testing. Think about it early, because it is part of making interoperability work in practice. 

Miguel Diez Blanco: I agree with Costas. In Europe, interoperability rarely fails on paper — it fails in implementation. The role of the Test Bed is to bridge that gap, by turning specifications into something that can be tested, trusted and reused. And once you take testing seriously, you shouldn’t have to build it yourself but focus on your core business and use what already exists. That’s exactly where the Test Bed comes in.