Linked Data Event Streams (LDES) applies - as the term implies - the concept of linked data to data streams. LDES is a technical standard that allows data to be exchanged between silos sustainably and cost-effectively using domain-specific ontology for fast and slowly changing data streams.
Introduction
Today, the main task of data publishers is to meet the user’s needs and expectations, and they realise this by creating and maintaining multiple querying APIs for their datasets. However, this approach causes various drawbacks for the data publisher. First, keeping multiple APIs online can be costly as the load generated by data consumers is often at the expense of the data publisher. And second, data publishers must ensure their APIs are always up-to-date with the latest standards and technologies, which results in a huge mandatory maintenance effort.
As new trends emerge, old APIs may become obsolete and incompatible, creating legacy issues for data consumers who may have to switch to new APIs or deal with outdated data formats and functionalities. Moreover, the existing APIs limit data reuse and innovation since the capabilities and limitations of these APIs constrain data consumers. They can only create their views or indexes on top of the data using a technology they prefer.
“Linked Data Event Streams as the base API to publish datasets”
On the other hand, data consumers often have to deal with multiple versions or copies of a dataset that need to be more consistent or synchronised. These versions or copies may have different snapshots or deltas published at other times or frequencies. Although using data dumps provides a data consumer with complex flexibility regarding the needed functionality, this approach also has various drawbacks. For example, data consumers must track when and how each version or copy was created and updated. They also have to compare different versions or copies to identify changes or conflicts in the data. Data consumers may need to be more consistent due to outdated or incomplete versions or copies. They may also miss significant changes or updates in data not reflected in their versions or copies.
To overcome these challenges, Linked Data Event Streams (LDES) provide a generic and flexible base API for datasets. With LDES, data consumers can set up workflow to automatically replicate the history of a dataset and stay in sync with the latest updates. Multiple researchers are investigating the opportunities that LDES provide starting from the basics, as described in this blog post of the Spanish Ministry of Digital Transformation.
Specification
The Linked Data Event Stream (LDES) specification (ldes:EventStream) allows data publisher to publish their dataset as an append-only collection of immutable members in its most basic form. Consumers can host one or more in-sync views on top of the default (append-only) view.
An LDES is defined as a collection of immutable objects, often referred to as LDES members.
These members are described using a specific format called RDF, which stands for Resource Description Framework. RDF is one of the corner stones of Linked Data and on which LDES continues to build.
The LDES specification is based on a hypermedia specification, called the TREE specification. The TREE specification originates from the idea to provide an alternative to one-dimensional HTTP pagination. It allows to fragment a collection of items and interlink these fragments. Instead of linking to the next or previous page, the relation describes what elements can be found by following the link to another fragment. The LDES specification extends the TREE specification by stating that every item in the collection must be immutable. The TREE specification is compatible with other specifications such as activitystreams-core, VOCAB-DCAT-2, LDP, or Shape Trees. For specific compatibility rules, please refer to the TREE specification.
LDESes apply — as the term implies — the Linked Data principles to data event streams. A data stream is typically a constant flow of distinct data points, each containing information about an event or change of state that originates from a system that continuously creates data. Some examples of data streams include sensor and other IoT data, financial data, etc.
Today, custom code has to be created to integrate data, which makes it rather expensive to integrate multiple data sources. With LDES, a technical standard was created that allows data to be exchanged across silos using domain-specific ontologies. An LDES allows third parties to build specific services (WFS, SPARQL endpoint) themselves on top of their own database that is always in sync with the original dataset.
An LDES is a constant flow of immutable objects (such as version objects of addresses, sensor observations or archived representations) containing information changes that originates from a system that continuously creates data. Compared to other event stream specification, the LDES specs opts to publish the entire object for every change.
Furthermore, LDES increases the usability and findability of data, as it comes in a uniform Linked Data standard published on an online URI endpoint. As a result, an LDES is self-descriptive meaning and more data can be found by following the links.
In a nutshell, there are several reasons why there was a need to develop the Linked Data Event Streams specification:
- Linked Data is a powerful paradigm for representing and sharing data on the Web. Still, it has traditionally focused on representing static data rather than events or changes to that data.
- The use of event streams is becoming increasingly prevalent on the Web, as it enables applications to exchange information about changes to data in real-time efficiently.
- There was a need for a semantic standard that provides a uniform way to exchange data so that different systems could easily exchange data.
- Linked Data Event Streams allow applications to subscribe to a stream of data and receive updates in real-time.
LDES Implementation - Challenges and Benefits
Linked Data Event Streams could be used to maintain and open up reference datasets to foster interoperability by advocating the reuse of the identifiers for which they are the authoritative source. By making data accessible via one URI endpoint, LDES makes it easy to retrieve data which is published as a uniform linked data standard.
This way, they tackle two often occurring challenges while trying to stay in sync with a data source.
- On the one hand, data publishers could provide periodical data dumps or version materialisations, which users have to fully download to stay up to date with the dataset.
- With a querying API, on the other hand, users can query the dataset without first having to download the entire dataset.
Trying to meet the needs of their reusers, data publishers will have to provide and maintain an increasing amount of such querying APIs as specific end-user features are solved by creating feature specific APIs.
However both data dumps and querying APIs will never fully meet the needs of their end-users, as a data dump gives a possibly outdated view on the dataset, whereas a querying API provides its client only with a partial view of the dataset.
LDES Implementation - Getting Started
To get started with the implementation of LDES, the following sections will delve into Flanders' (BE) approach, which they are willing to share with the rest of Europe. This section is divided in both LDES CLIENT, for replication and synchronisation, and LDES SERVER for publishing one or multiple Linked Data Event Stream(s).
LDES CLIENT
The LDES CLIENT is designed for replication and synchronisation, meaning the client can retrieve members of an LDES but also checks regularly if new members are added and fetch them, allowing data consumers to stay up to date with the dataset.
To understand the functioning of an LDES client, it is important to understand how LDESes are published on the Web. The Linked Data Fragments principle is utilised for publishing an LDES, meaning that the data is published in one or more fragments and meaningful semantic links are created between these fragments. This approach facilitates clients to follow these links and discover additional data. However, the critical aspect for the LDES client is the notion of mutable and immutable fragments. When publishing an LDES stream, a common configuration is to have a maximum number of members per fragment. Once a fragment surpasses this limit, it is regarded as immutable, and a ‘Cache-control: immutable’ cache header is added to the fragment to signify this. This information is crucial for the LDES client since it only needs to retrieve an immutable fragment once, while mutable fragments must be regularly polled to identify new members.
More information about the replication and synchronisation of an LDES can be found on the specific documentation page of Digital Flanders.
LDES SERVER
The Linked Data Event Stream (LDES) server is a configurable component that can be used to ingest, store, and (re-)publish one or multiple Linked Data Event Stream(s). The open-source LDES server is built in the context of the VSDS project to exchange (Open) Data easily.
The server can be configured to meet the organisation’s specific needs. Functionalities include retention policy, fragmentation, deletion, create a snapshot and pagination for managing and processing large amounts of data more efficiently and ensuring the efficient use of storage.
For more information about setting up the LDES Server, have a look at this page.
How can SEMIC support?
SEMIC can provide support for all Member States or organizations who would like to gain insights on how a LDES might be a solution for their problem.
- Implementation pilots: SEMIC provides the specifications on LDES as the main starting point for all interested parties. Our team of experts can also provide further support by collaborating with the public administrations technical teams, to ensure optimal implementation.
- Strategy and roadmap: We can support in identifying possible use cases in your organisation and help you in defining a set of concrete implementation steps to progressively and effectively adopt LDES.
- Knowledge sharing: SEMIC can provide knowledge sharing sessions in the form of webinars or even 1:1 roadshows in which key stakeholders are given the chance to exchange experiences and best practices:
Please feel free to reach out to emiel.dhondt@pwc.com if you have any questions on the topic on LDES.
SEMIC TV
Interested in the linked data event streams and how SEMIC contributes to it? Here is what Pieter Colpaert, Professor at IDLab (Ghent University) would like to share with you!
Access LDES specifications in SEMIC GitHub
