From interoperability to knowledge discovery using large model datasets in the marine environment: the THREDDS Data Server example

In order to monitor, describe and understand the marine environment, many research institutions are involved in the acquisition and distribution of ocean data, both from observations and models. Scientists from these institutions are spending too much time looking for, accessing, and reformatting data: they need better tools and procedures to make the science they do more efficient. The U.S. Integrated Ocean Observing System (US-IOOS) is working on making large amounts of distributed data usable in an easy and efficient way. US-IOOS is essentially a network of scientists, technicians and technologies designed to acquire, collect and disseminate observational and modeled data resulting from coastal and oceanic marine regions investigations to researchers, stakeholders and policy makers. In order to be successful, this effort requires standard data protocols, web services and standards-based tools. Starting from the US-IOOS approach, which is being adopted throughout much of the oceanographic and meteorological sectors, we describe here our experience with the approach at CNR-ISMAR Venice using the THREDDS (THematic Real-time Environmental Distributed Data Services) Data Server (TDS), middleware designed to fill the gap between data providers and data users. The TDS provides services that allow data users to find the data sets pertaining to their scientific needs, to access, to visualize and to use them in an easy way, without downloading files to the local workspace. In order to achieve this, it is necessary that the data providers make their data available in a standard form that the TDS understands and with sufficient metadata to allow the data to be read and searched in a standard way.The core idea is to utilize the Unidata Common Data Model (CDM), a unified conceptual model that describes different datatypes within each dataset. Unidata (www.unidata.ucar.edu), supported by the US National Science Foundation, has developed CDM specifications for many of the different kinds of data used by the scientific community, such as grids, profiles, time series, swath data. These datatypes are aligned the NetCDF Climate and Forecast (CF) Metadata Conventions and with Climate Science Modelling Language (CSML). This means that CDM objects can be constructed as virtual NetCDF datasets from physical NetCDF, GRIB or HDF files, and if necessary, modified or supplemented with NetCDF Markup Language (NcML) through XML to meet requirements for metadata. CF-compliant NetCDF files and GRIB files can be read directly with no modification. Once standardized in the CDM, the TDS makes datasets available through a series of web services such as OPeNDAP or Open Geospatial Consortium Web Coverage Service (WCS), allowing the data users to easily obtain small subsets from large datasets, and to quickly visualize their content by using tools such as GODIVA2 or Integrated Data Viewer (IDV). In addition, an ISO metadata service is available through the TDS that can be harvested by catalog broker services (e.g. GI-cat) to enable distributed search across federated data servers. Example of TDS datasets can be accessed at the CNR-ISMAR Venice site http://tds.ve.ismar.cnr.it:8080/thredds/catalog.html.