We as conterra work on the European Data Portal. One main target (beside the provision of a map.apps based mapClient and a Gazetteer) is the harvesting of metadata from European geospatial metadata catalogues. Usually those catalogues do not support CKAN/DCAT-AP (as required by the standard EDP harvesters) but rather community specific interfaces as defined by INSPIRE regulations [1][2], the Open Geospatial Consortium (OGC)[4][7][8] or Copernicus/HMA (Heterogeneous Missions Access) [5][6]. Most important in this context are the INSPIRE related metadata catalogues.

In the EU Open Data Portal the Harvester is a separate component that is able to harvest data from different data sources with different data formats and APIs. It is able to transform each harvested metadata into CKAN JSON schema. The harvester uses a specialized connector-type for the geo portals, called Geo-harvester, providing one well defined interface. This interface (aka Open Data Geo - ODG) is a subset of the OAI-PMH harvesting protocol specification [11]. It provides the data in DCAT-AP format. It is further able to process the harvesting in blocks (partitioning) and supports differential harvesting based on the metadata´s datestamp. The ODG Interface is based on HTTP/GET/REST. The Interface accepts simple „query-“ (or better harvesting-) requests with just a few parameters (e.g. from (startDateTime), until (endDateTime) (for differential harvesting, if supported) and startPos/maxRecs (resumption, for partitioning).

Different Geo-harvester implementations exist, one for every geospatial metadata catalogues interface type.
The mapping from the response of the geospatial metadata catalogue to DCAT-AP response was the most important task done. This is based on XSLT. We had to deal with different challenges, e.g. the mapping of metadata elements provided as character strings (e.g. keywords) into links of existing vocabularies/ontologies. An example here is the linking to an INSPIRE Annex Theme (URI) of the GEMET thesaurus. The alignment of
INSPIRE metadata with DCAT-AP is defined in GeoDCAT-AP [21]. Here you´ll find a core and an extended version. The core version defines alignments only for the subset of INSPIRE metadata elements that can be mapped to the ones included in DCAT-AP. The alignments for those INSPIRE metadata elements not included in GeoDCAT-AP Core are defined in the extended version of GeoDCAT-AP called GeoDCAT-AP Extended [21].

A sample request may look like: /oai-pmh-geo-harvest/gdi-de?
from=20140101T00:00:00&until=20140101T23:59:59&resumption=51:dcat_ap::

The requests are able to return EU DCAT-AP Core and Extension [13][12] metadata format (if request succeeded).
At the time of writing CSW/INSPIRE-POST- [1], CSW/INSPIRE-SOAP1.1/1.2- [1] and OpenSearchGEO/EOP-REST [8][22] Adapters are implemented.

The adapters are able to do the request- and response protocol conversion, e.g. ODG -> CSW/INSPIRE request and CSW/INSPIRE ISO19139 responses to ODG-DCAT-AP.. In a later stage harvesters may support additional interfaces (e.g. [5],[6], [9], [10]). They further provide differential harvesting and partitioning.
For every Geo-Catalogue Instance which has to be harvested and which provides one of the interfaces for which an adapter exists, one instance is instantiated (providing a specific REST-URL: e.g. //oai-pmh-geo-harvest/harvesters/gdi-de? for the GDI-DE Catalogue).
CKAN just need to provide one type of harvester: the ODG Harvester which implements an ODG Client. This will be instantiated once for every Geo-Catalogue.

Currently the source is provided here: https://gitlab.com/EU_ODP/geo-harvesting (password protected for the Moment, will be open source in the near future).

REFERENCES:
[1] Technical Guidance for the implementation of INSPIRE Discovery Services, Initial Operating Capability Task Force for Network Services, 07-11-2011
[2] INSPIRE Metadata Implementing Rules: Technical Guidelines based on EN ISO 19115 and EN ISO 19119, European Commission Joint Research Centre, 16-06-2010
[4] OGC Catalogue Services Specification 2.0.2 - ISO Metadata Application Profile, Version 1.0, OGC doc 07-045
[5] OGC I15 (ISO19115 Metadata) Extension Package of CS-W ebRIM Profile, version 1.0, OGC 13-084.
[6] INSPIRE Conformance Class of OGC I15 (ISO19115 Metadata) Extension Package of CS-W ebRIM Profile - I15 EP Protocol Binding of INSPIRE Discovery Services -. OGC Engineering Report. OGC document 08-197r4.
[7] OGC Catalogue Services Specification 2.0, with Technical Corrigendum 2, OGC 07-006r1
[8] OGC OpenSearch GeoSpatial and Temporal Extensions, version: 1.0.0, OGC 10-032r6
[9] SRU - Search/Retrieval via URL (http://www.loc.gov/standards/sru/)
[10] WIS Compliance Specifications of GISC, DCPC and NC, v1.2, http://www.wmo.int/pages/prog/www/WIS/documents/TechnicalSpecification1…
[11] The Open Archives Initiative Protocol for Metadata Harvesting. http://www.openarchives.org/OAI/openarchivesprotocol.html.
[12] Data Catalog Vocabulary (DCAT). http://www.w3.org/TR/vocab-dcat/
[13] DCAT application profile for data portals in Europe. https://joinup.ec.europa.eu/asset/dcat_application_profile/asset_releas…
[16] http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32008R1205…
[17] http://opendefinition.org/od/
[18] ISO 19115:2003, Geographic information Metadata (with ISO 19115:2003/Cor.1:2006, Geographic information - Metadata -Technical Corrigendum 1)
[19] ISO-DIS 19115-2, Geographic information - Metadata - Part 2: Extensions for imagery and gridded data, 2007
[20] ISO/TS 19139 (10/2005), Geographic information - Metadata -Implementation specification
[21] Specification of GeoDCAT-AP. GeoDCAT-AP: a geospatial extension for the DCAT application profile for data portals in Europe. European Comission. Version 1.0. https://joinup.ec.europa.eu/asset/dcat_application_profile/asset_releas…
[22] OGC OpenSearch Extension for EarthObservation, version: 1.0.0, OGC 13-026

DG CONNECT has developed a DCAT-AP validator which should comply with DCAT-AP 1.1 specification and it is released as open source software under EUPL licence.

In practice the validator stores the RDF file (coming from different sources: file upload, URL, direct input) into the triple store connected and validates the graph created against the SPARQL query which is the union of the rules coming from the specification.

Currently the validator doesn't have an API which would allow to validate the RDF file programmatically during the harvesting phase and it would be nice to extend it to GeoDCAT-AP and GeoDCAT-AP Extended.

You may be interested in: 

- https://github.com/LifeWatchINBO/ipt-dcat (GBIF / Biodiversity)

- http://thedatatank.com/ (portal software for publishing / transforming data, the homepage mentions it supports DCAT-AP)

Proposed resolution:

• Call for information for tools in various categories:
  • Editors (allowing for manual input and/or input validation)
  • DCAT-AP exporters
  • Mapping of themes
  • DCAT-AP validators

After the online meeting of Friday 18 March, the working group concluded on the following categories of tools for the supporting DCAT-AP implementations: 

• Editors (allowing for manual input and/or input validation)
• DCAT-AP exporters
• Mapping of themes
• DCAT-AP validators
• DCAT-AP harvesters

Members of the working group who have developed tools for the DCAT-AP are invited to complete basic information about their tools here. 

Possibly relevant tools concerning GeoDCAT-AP (some of them already mentioned by Uwe):

GeoDCAT-AP XSLT: This XSLT implements the mappings defined in the GeoDCAT-AP specification, and can be used to transform INSPIRE / ISO 19115 records into DCAT-AP or GeoDCAT-AP.

The GeoDCAT-AP XSLT (and the corresponding documentation) is available from the Commission's Git repository:

https://webgate.ec.europa.eu/CITnet/stash/projects/ODCKAN/repos/iso-19139-to-dcat-ap/

GeoDCAT-AP API: This API is a proof-of-concept of the implementation of GeoDCAT-AP in an OGC CSW (Catalog Service for the Web), re-using the standard CSW interface, and supporting in addition HTTP content negotiation.

The API uses the GeoDCAT-AP XSLT to transform ISO 19139 records into GeoDCAT-AP. As such, the API works both on static files including the records, and on the CSW output of a "GetRecords" or "GetRecordById" request.

A working demo of GeoDCAT-API is available at:

http://geodcat-ap.semic.eu:8890/api/

The source code of the GeoDCAT-AP API (and the corresponding documentation) is available from the Commission's Git repository:

https://webgate.ec.europa.eu/CITnet/stash/projects/ODCKAN/repos/iso-19139-to-dcat-ap/browse/api

Mapping of INSPIRE themes and ISO topic categories to the MDR Data Themes: Work is under-way in the framework of the GeoDCAT-AP WG for defining mappings between the INSPIRE themes and the ISO topic categories (i.e., the controlled vocabularies used in INSPIRE / ISO 19115 metadata) to the MDR Data Themes.

The alignments are specified by using SKOS, and encoded in RDF/XML. They are available from the Commission's Git repository:

https://webgate.ec.europa.eu/CITnet/stash/projects/ODCKAN/repos/iso-19139-to-dcat-ap/browse/alignments

An HTML preview is made available via the instance of the SKOS-Matches Viewer API at the following URL:

http://geodcat-ap.semic.eu:8890/skos-match-viewer/