Virtual Hubs for facilitating the use of geospatial datasets from heterogeneous data sources

The world of geoinformation is characterized by a great complexity with many actors involved including: Data (and information) producers who acquire observations (e.g. through sensors) or generate value-added information (e.g. through data processing); Data providers who distribute data, managing data centres, long-term preservation archives, Spatial Data Infrastructures, etc.; Overarching initiatives that influence the geoinformation world designing new solutions, building disciplinary or interdisciplinary systems of systems, managing high-level expert groups, etc.; Technology providers who develop and distribute technological solutions for geospatial data management and sharing; Application developers who make use of data to build applications for end-users; End-users who utilize data. In such a context, interoperability is perceived as a main issue even limiting to technological aspects. Indeed, all of actors have a substantial impact in terms of technological choices: Data (and information) producers are mostly focused on data and metadata models and formats defining specifications such as HDF, netCDF and GRIB for EO data, ESRI Shapefile or OGC GML for feature type information; Data providers are mainly focused on data sharing services defining and adopting solutions like OGC discovery and access standards, TDWG specifications, THREDDS data servers; Overarching initiatives influence technological aspects in several ways, in particular on data, data harmonization, and data sharing including policy; Technology providers contribute to the heterogeneity providing many different competing solutions for geospatial data sharing; Application developers increase the heterogeneity because they provide geospatial applications adopting different technologies, from operating systems and related ecosystems (e.g. Linux, Microsoft, Apple, Google Android), to development platforms (e.g. Java, Python, Javascript) and libraries. Lack of interoperability is then perceived as one of the main barriers to data sharing and use: a potential user, being he either an intermediate user like an application developer, or an end user, must be or become an expert in geospatial technologies to be able to discover, access and use geospatial data served by different systems and provided in different models and formats. In October 2014, the ENERGIC OD (European NEtwork for Redistributing Geospatial Information to user Communities - Open Data) innovation project, funded by the European Union under the Competitiveness and Innovation framework Programme (CIP) has started aiming at the "development of virtual hubs that facilitate the use of open (freely available) geographic data from different sources for the creation of innovative applications and services". ENERGIC OD designed and implemented a set of virtual hubs based on the innovative approach of brokered architectures firstly adopted in building large infrastructures in the context of the Global Earth Observation System of Systems (GEOSS) to overcome limitation of the federated approach. The notion of "System of Systems" (SoS) and "System of Systems Engineering" (SoSE) emerged in many fields of applications to address the common problem of integrating many independent, autonomous systems [1]. From a technical point-of-view, there are two general approaches for building a SoS: through federation and through brokering. In the federated approach, the participants agree on a common set of specification (federated model) rangin from the adoption of a suite of interfaces, metadata and data models, to a very strict approach imposing the adoption of the same software tools. To be successful, this approach requires: a) the definition of the common model, which can be very complex for a big multidisciplinary SoS; b) the adoption of the common model by participants, which may need major changes in the existing systems; c) the enforcement of compliancy with the common model which needs a governing body. The brokered approach overcomes those difficulties, since no common model is defined, and participating systems can adopt or maintain their preferred interfaces, metadata and data models. Instead, specific components - i.e. the brokers - are in charge of accessing the participant systems, providing all the required mediation and harmonization functionalities [2]. Obviously, the inherent complexity of mediation process does not disappear in the brokering approach. It is simply moved to the brokers which are necessarily complex components. Although the development of brokers can be a big technological challenge, the brokering approach conforms with the separation-of-concern engineering pattern, setting free data providers and data users from any interoperability concern. An ENERGIC OD Virtual Hub is then conceived as a virtual node where users can seamlessly access potentially unlimited datasets by brokering heterogeneous open geospatial data sources. Making heterogenous data accessible through one or more uniform interfaces is not sufficient to facilitate the use of geospatial information. Geospatial information is inherently complex, including different feature types, requiring complex models and formats. Moreover, many user requirements need to be supported making common standard interfaces necessarily complex. However, a large number of applications do not require to handle the full complexity of the geospatial information, accepting the reduction to simpler models and the use of simplified interfaces providing the main functionalities. For this reason, ENERGIC OD Virtual Hubs, beside full standard discovery and access interfaces, provide simple Application Programming Interfaces (APIs) easily usable by developers who are not expert of geospatial science and technology. The ENERGIC OD project developed a Virtual Hub software platform which can be deployed on local infrastructures or on private or public clouds, also supporting elastic capabilities for big data handling. It is based on an existing and extensible mature brokering technology, the GI-Suite Brokering Framework by CNR-IIA, which is also adopted for the development of the GEO Discovery and Access Broker (DAB), a central component of the GEOSS Common Infrastructure (GCI) [3]. During the ENERGIC OD project five national-level Virtual Hubs have been deployed in France, Germany, Italy, Poland and Spain, and one local-level Virtual Hub has been deployed in Berlin serving the metropolitan area. Each Virtual Hub collects data relevant for the development of desktop and mobile apps in the interested geographical area. Dataset provided through the Virtual Hubs include satellite data (including Landsat and Sentinel), geographical information from public authorities, volunteered geographic information from crowdsourcing, etc. Ten pilot applications with different geographical and thematic coverage have been successfully developed to demonstrate the validity of the ENERGIC OD approach and implementation. The outcomes of the ENERGIC OD project demonstrate the feasibility and efficiency of the virtual hub approach when there is the necessity of integrating datasets coming from heterogeneous sources and to allow developers to create applications without the need to be an expert of geospatial technologies. [1]M. Jamshidi, System of Systems Engineering - Innovations for the 21st Century. John Wiley & Sons, 2009. [2]S. Nativi, M. Craglia, and J. Pearlman, "Earth Science Infrastructures Interoperability: The Brokering Approach," 2013. [3]"Virtual hubs - system architecture second release - Deliverable D5.1." 2016.