CEOP Coordinated Energy and water cycle Observations Project

Data Management (DM)

Data Management (http://www.eol.ucar.edu/projects/ceop/dm/), a former key component of both GHP and then 'CEOP', a key component of CEOP, which has successfully managed to get diverse international groups to agree to a general data policy and other groups to then help maintain an internationally distributed database of extensive hydrometeorological data.

The success of CEOP data management has been in linking to the various RHPs and contributing national agencies to develop in-situ, model output and satellite remote sensing archives and providing the data to the science community. The three CEOP Archive Centers are: (1) the NCAR/Earth Observing Laboratory (EOL) Central Data Archive for CEOP Reference Site Data; (2) the International Council for Science (ICSU) WDC for Climate which is supported by the Max Planck Institute for Meteorology (MPIM) and the German Climate Computing Center (DKRZ) in Hamburg for NWP Model Output; and (3) the Satellite and Data Integration Center at the University of Tokyo (UT), which is jointly administered by the Japan Aerospace Exploration Agency (JAXA), the Japan Meteorological Agency (JMA), and UT. These Centers abide by the CEOP Data Policy Criteria described below, established with the data providers and this data is then freely available to any and all users free of any costs, with the exception of possible modest expenses for mailing costs of any data that is requested other than through the electronic schemes set up by the administers of each of these centers. The data users are required to comply with the use and citing of the material in accordance with the CEOP data policies as noted for the in-situ data, the model data and the satellite data.

Data Policy

Improvement of understanding and predictability of the energy and water cycle cannot be achieved without data and information sharing including data management and integration that encompasses functions such as Quality Assessment/Quality Control, access to data, and archiving of data, data integration and visualization, and information fusion.

The WMO Resolutions 40 and 25 (adopted by the XII Congress on 26 October 1995) comprises the basis for the CEOP data policy and protocol to be adopted and practiced by each of the CEOP Data Archive Centers:

"As a fundamental principle of the World Meteorological Organization (WMO), and in consonance with the expanding requirements for its scientific and technical expertise, the WMO commits itself to broadening and enhancing the free and unrestricted international exchange of meteorological and related data and products".

In general, the scientific community will have free and open access to all CEOP data, subject to procedures in place at the various Data Archive Centers involved. The following six "golden rules" have been established as part of the CEOP Data Policy to ensure a smooth and successful use of CEOP Data by the scientific community:

  • No financial implications are involved for the CEOP reference site data exchange.
  • Commercial use and exploitation of CEOP reference site data is prohibited.
  • Any re-export or transfer of the original data received from the CEOP archive to a third party is prohibited.
  • The origin of CEOP data being used for publication of scientific results must be acknowledged and referenced in the publication.
  • CEOP data users are strongly encouraged to establish direct contact with data providers for complete interpretation and analysis of data for publication purposes.
  • Co-authorship of data users and CEOP data sources. Principle Investigators on publications making extensive use of CEOP data is justifiable and highly recommended.

Further details regarding the CEOP Data Policy may be found at: http://www.eol.ucar.edu/projects/ceop/dm/documents/ceop_policy.html.

Top of this page

Home


Reference Site/Basin Data Archive

http://www.eol.ucar.edu/projects/ceop/dm/ (Reference Site Data Gateway)

Chair: Steve Williams (sfw@ucar.edu)

During the Coordinated Enhanced Observing Period ('CEOP') key agreements were initiated to obtain in-situ data from 35 selected globally distributed "referenceh stations (Figure 1) involved with the various CEOP RHPs. These Reference Sites provide enhanced observations of sub-surface (soil profiles), surface (standard meteorological and radiation), near surface (flux tower), atmospheric profiles (rawinsonde and profiler), and ancillary data sets (radar, special observations) in a common format. All Reference Site data are archived and disseminated by the NCAR/EOL, in Boulder Colorado. The Reference Site Data Archive ensures that each request for data from the CEOP Reference Site Archive is accompanied, automatically/electronically, by a copy of the CEOP Data Policy.

Collection of the data from the CEOP Reference sites for the 'CEOP' period (2001 - 2004) has shown that adherence by the sites to a consistent format is especially important to ensure an efficient continuation of the CEOP dataset development and delivery process. The reference site characteristics and metadata have been cataloged along with all of the information about the entire CEOP in-situ database. The data collection has been an on-going process and it is the role of the CEOP Data Management Working Group to maintain the status of the contributions from the CEOP reference sites at the RHPs.

For the period of Coordinated Energy and Water Cycle Observations Project (2007 - 20011), the original set of 35 'CEOP' Reference Sites has been augmented by including a number of new sites, which reflects recent commencement of several RHPs (e.g. LPB, MAHASRI) and establishment of CEOP Regional Studies. Also, a few sites were excluded from the original 'CEOP' set due to the termination of observation at those sites or inability to continue to fulfill the CEOP requirements for the Reference Sites. The updated set of the CEOP Reference Sites is shown in Figure 2.

Global distribution of 35 'CEOP' Reference Sites
Global distribution of 51 CEOP Reference Sites

Figure 1: Global distribution of the 'CEOP' 35 Reference Sites. (Click on the image for a higher resolution figure).

Figure 2: Global distribution of the CEOP 52 Reference Sites. (Click on the image for a higher resolution figure).

Top of this page

Home


Model Output Archive

http://www.eol.ucar.edu/projects/ceop/dm/model/

http://ceop.wdc-climate.de (Model Output Data Gateway)

Chair: Michael Lautenschlager (michael.lautenschlager@zmaw.de)

Ten operational Numerical Weather Prediction (NWP) and two data assimilation centers are currently contributing analysis/assimilation and forecast model products from global and regional NWP suites, including both operational and reanalysis systems to this component of CEOP. The contributing centers include:

  • BoM: Bureau of Meteorology
  • CPTEC: Centro de Previsao de Tempo e Estudos Climaticos
  • ECMWF: European Centre for Medium-Range Weather Forecasts
  • ECPC: Experimental Climate Prediction Center
  • EMC: EPSON Meteo Center (Centro EPSON Meteo)
  • GLDAS: Global Land Data Assimilation System
  • GMAO: NASA Global Modeling and Assimilation Office
  • JMA: Japan Meteorological Agency
  • MSC: Meteorological Service Canada
  • NCEP: National Centers for Environmental Prediction
  • NCMRWF: National Center for Medium Range Weather Forecasting
  • UKMO: UK Met Office

The Max-Planck Institute for Meteorology (MPIM) in coordination with the ICSU World Data Center for Climate (WDCC) in Hamburg, Germany was designated as the CEOP model output archive center. The WDCC is administered by the Model and Data Group (M&D) at MPIM and the German Climate Computing Center (DKRZ).

To assist with the organization of this activity during the Coordinated Enhanced Observing Period ('CEOP'), a Model Output Management Document was drafted as a guide for the participating centers to use in setting up their processes for meeting their commitments to 'CEOP'. The Guidance Document addressed the two issues of (1) the model output variables requested by 'CEOP' and (2) the two types of requested model output, namely global gridded (in GRIB format) and site-specific Model Output Location Time Series (MOLTS) at each of the 'CEOP' Reference Sites.

A new version of the Guidance Document will be compiled that clarifies what model output data will be generated by the NWP Centers and Groups contributing to the model output component of Coordinated Energy and Water Cycle Observations Project (CEOP) and how they will interface/transfer the data that will be handled and retained at the WDCC. The issues covered in the document will include: (1) global versus regional products; (2) desired assimilation output; Interval and length of free-running forecasts; (3) Operational versus reanalysis data; (4) the CEOP schedule/archive periods; (5) the number and locations of MOLTS sites; and (6) the homogenizing of the model output and metadata formats (i.e. standard parameters).

Results up to this point in the CEOP model output generation effort make it clear that the transfer aspect of the data handling effort has been progressing well. Data from all twelve Centers participating in CEOP have been received at the data archive center and has either been placed into the database at the Hamburg facility, or is in the process of being entered into the database. The current data holdings in the MPIM archive can be viewed here.

Top of this page

Home


Satellite Data Archive

http://monsoon.t.u-tokyo.ac.jp/ceop2/satellite/ (Satellite Data Gateway)

Chair: Toshio Koike (tkoike@hydra.t.u-tokyo.ac.jp)

It is essential to integrate the data from satellites observing both land and ocean in generating new CEOP data sets for the overall water cycle. The work associated with satellite data set archive is being undertaken by the University of Tokyo in collaboration with JAXA, NASA, ESA, and NOAA. The Committee on Earth Observation Satellites (CEOS) membership encompasses the worldfs government agencies responsible for civil Earth observation satellite programs. Within CEOS, the Working Group on Information Systems and Services (WGISS) aims to coordinate and monitor the development of the systems and services, which manage and supply the data and information from participating organizationsf missions. Under this coordination framework, NASA, ESA, and NOAA are providing their satellite data sets to the UT archiving system. In addition to providing their own data, JAXA works as a coordinator of the CEOP satellite data archive in CEOS/WGISS.

The data set consists of the main water cycle parameters necessary to accomplish CEOP scientific goals; these data are geo-coded (i.e. re-sampled to a regular LAT/LON Grid). They are generated at three scales, 250km rectangular, monsoon regional and global scales, associated with product levels 1b, 2 and 3. The processing levels have also been defined to ensure a clear understanding of the nomenclature and reduce ambiguity in the statement of requirements. The Levels of processing have been established to be:

  • [Level-1b] - Radiance product with full resolution at reference sites.
  • [Level-2] - Geophysical product at the same resolution at reference sites and monsoon regions.
  • [Level-3] - Statistical geophysical product in space and/or time at reference sites, monsoon regions and global. (example: Monthly mean rain rate at reference sites, etc.)

These components consist of an image element and a metadata part element that is compliant with the ISO-19115 metadata standard.

Top of this page

Home


Data Integration and Dissemination

Chair: ()

Data Integration

As originally produced by the various sources, the data is in a wide variety of formats and structures. In response to this situation there was recognition of need for data management systems for the collection, sharing and provision of data from which users can obtain precisely the data they need, whenever they want it and in formats familiar to the science community. It is essential to transform observation data into scientifically and socially relevant information through the systematic collection and integration of data, merging of essential related information, and building of systems for sharing this knowledge on an international basis.

In cases in which large amounts of heterogeneous observation data are handled, this data along with socioeconomic and other related data must be dealt with systematically in order to produce useful scientific knowledge and translate it into information pertinent to users. Communicating comparatively lighter amounts of heterogeneous data requires the standardization of data formats and protocols, and the development of a network-linked distributed data system.

Interoperability and Metadata

Metadata is commonly defined as "data about data", which may provide information about the content, quality, condition, and characteristics of data. Major applications of metadata require CEOP to:

  • Organize and maintain data in which significant resources may have been invested.
  • Provide information to data catalogs and data clearinghouses.
  • Provide information to aid data transfer.

Standardized metadata can provide data producers with the format and content needed to properly describe their data, and to effectively and efficiently manage and maintain data. Also, it can support data users in finding and obtaining useful data more effectively and efficiently. The ISO TC/211 19115 metadata standard defines the schema required for describing geographic information and services and provides information about identification, extent, quality, spatial and temporal schema, spatial reference and distribution of digital geographic data. CEOP metadata was developed in conformity to this ISO metadata standard.

An implementation approach has been introduced for metadata application to CEOP data services over networks.

  • Clients perform service discovery operations on the registry service to find the service providers it needs and then accesses service providers for provision of the desired service.
  • Registry services help service providers and clients to find each other by acting as a registry or clearinghouse of services.
  • Service providers publish services to a registry and deliver services to clients.

To improve data inter-operability, an ontology system development is now on going for supporting target data and information from diverse data sources with high complexity. Data integration needs to find meaning similarity and difference among different disciplines. To find data with similar meanings, it is often necessary to confirm the definition of the terminology. The ontology system teaches the definition. Furthermore, when the name of data is not clear, the system can provide several candidate data name by inputting keywords into the system. Standardized metadata should be prepared for data exploring, confirmation of data meaning and quality, and open data sharing.

Top of this page

Home

Distributed Integrated Data Archive

http://jaxa.ceos.org/wtf_ceop/

To meet various CEOP science objectives, the CEOP science community requires data integration services that allow it to access and intercompare diverse data types from multiple sources. The WGISS agenciesf prototypes offer a variety of capabilities towards this goal that are accessible over the internet.

JAXA is developing a prototype distributed data integration system to provide user-friendly services for discovery, analysis and visualization of CEOP (in-situ, satellite and model output) data to water cycle research scientists globally. The prototype system provides users with menus for selection of data and services through a standard web browser, and for access to the CEOP data from the 3 CEOP data archive centers. The system is distributed in the sense that while the main server is located in Tokyo, the data is located in archive centers, which are globally distributed. Further, a global distribution of users is supported.

The system is integrated in multiple ways. First, while the data may be created and archived in a variety of formats the system masks the data formats and presents the data in a uniform style. Second, the system knows the geolocation and time of all the data sets, and coordinates the selection, retrieval and display of the various types of data both temporally and geospatially. Thirdly, the system supports selection of the data through a uniform set of menus, by data type, data providing agency, reference site and station, and supports sub-setting according to time, area and height/depth. The system provides user-friendly services to view plots and graphs of the data, to view data values on the screen, to provide information (metadata) about the CEOP data, or to download data from an archive directly to the userfs (local) computer.

The NASA team prototype is focusing on the enhancement of access to its satellite data resources by implementing access to satellite data through the OGC Web Coverage Service (WCS) and tailoring the OPeNDAP and WCS software to enable access to the WCS served satellite data by OPeNDAP enabled analysis clients. These OPeNDAP enabled analysis clients are used by many CEOP scientists. Enabling this access will enable CEOP scientists to more easily access and use satellite data with model and reference site data in a variety of analysis clients such as GrADS, Matlab, IDL, Ferret, LAS, Giovanni, etc. The NASA team plans to make their software available to other interested CEOS WGISS agencies that are interested in offering their agency satellite data through this mechanism. In the future, additional data services in support of data integration may be selected to be offered through the NASA software.


Centralized Integrated Data Archive

http://monsoon.t.u-tokyo.ac.jp/ceop-dc/ceop-dc_top.htm

Chair: Kenji Taniguchi (taniguti@hydra.t.u-tokyo.ac.jp)

Diverse and large-volume Earth Observation data from inhomogeneous information sources is archived by a 1 Peta-Byte hard disk array and a complementary tape library. Some analysis operations such as average, difference, correlation and visualization can be applied to single or multiple data types through the interface. The browse and analysis interface is performed by dedicated clients, which provide the users with menus, integrated access to the data, and analysis tools. The connection between the clients and the server is based on HTTP. Users can access all types of data through a single interface and can view the retrieved data as graphic charts or bitmap images, depending on their dimensionality. User can integrate various observed data and numerical model outputs easily and effectively. Identifying targeted data, date, and region, the user can obtain analyzed outputs and then visualize the results on a display wall. 3D visual analysis tools are also available.

Top of this page

Home


Associated Data Centers

There are a number of additional data sources that will be contributing to CEOP including a number of Global Data Centers. The Global Runoff Data Center and the Global Precipitation Climatology Center have had long-term affiliation with GEWEX and GHP and will continue this collaboration under the new CEOP.

Global Runoff Data Center (GRDC)

http://grdc.bafg.de/

Chair: Ulrich Looser (looser@bafg.de)

The Global Runoff Data Centre (GRDC) was established in 1988 at the Federal Institute of Hydrology (BfG) under the auspices of the WMO. It is a contribution of the Federal Republic of Germany to the World Climate Programme Water (WCP-Water) of WMO. WMO mandates and directly supports GRDC by its Resolutions 21 (Cg XII, 1995: Request to the member states to provide GRDC with river discharge data) and 25 (Cg XIII, 1999: Free and unrestricted exchange of hydrological data). An international Steering Committee is guiding and overseeing the activities of the GRDC. The steering committee consists of representatives from WMO, UNESCO, UNEP, ICSU, IAHS and partner data centres GPCC and the International Groundwater Resources Assessment Centre (IGRAC).

The main objective of the GRDC is the worldwide acquisition, storage and dissemination of historical and near-real-time river-discharge data in support of the predominantly water and climate related programmes and projects of the United Nations (UN), their specialized agencies and the scientific research community. Also there are various GRDC activities in the framework of the GEWEX. GRDC is recognized as one of the hydrometeorological projects in GEWEX. As such the GRDC seeks to provide inputs to the GEWEX/CEOP RHP and modeling efforts by providing improved data sets.

As of end-2007 the GRDC database holds world-wide discharge data of 7,332 stations in 156 countries featuring around 276,000 station-years of monthly and daily values with an average time-series length of 37.7 years. Additionally the GRDC maintains the following four specialized databases:

  • Arctic Runoff Database (ARDB) containing data from over 2400 stations specifically for the arctic research community associated with the WCRP ACSYS (Arctic Climate System Study) and the CLiC Project
  • Terrestrial Network for River Discharge (GTN-R) database for near-real time data, a contribution towards the Implementation Plan for the Global Observing System for Climate and to GTN-H
  • European Terrestrial Network for River Discharge (ETN-R) database for near-real time data, a GRDC contribution towards the European Flood Alert System (EFAS) under development by the European Union (EU) Joint Research Centre
  • European Water Archive (EWA) in support of the Northern European Flow Regimes from International Experimental and Network Data (NE-FRIEND) research community

Global Precipitation Climatology Center (GPCC)

http://gpcc.dwd.de/

Chair: Tobias Fuchs (tobias.fuchs@dwd.de)

The Global Precipitation Climatology Centre (GPCC) was established in 1988 at the request of the WMO. The Deutscher Wetterdienst (DWD, National Meteorological Service of Germany) operates it as a German contribution to the WCRP. The mandate of GPCC is the global analysis of monthly precipitation on earthfs land-surface based on in situ rain gauge data. Since its start, the centre is the in situ component of the GPCP. In 1994, the long-term operation of the GPCC has been requested by WMO in order to contribute to the climate monitoring activities of the Global Climate Observing System (GCOS). Since 1999, GPCC has been a global GCOS Surface Network Monitoring Center (GSNMC), which has a special emphasis on precipitation. Mid December 2006, GPCC started its newest function as the WMO Commission for Basic Systems (CBS) Lead Center for GCOS data for Europe. The German Meteorological Service (DWD) continuously funds GPCC operation.

Main GPCC objective is the analysis of the spatial and temporal distribution of land-surface precipitation on a monthly time-scale based on in situ rain gauge data compiled from stations located worldwide. The aim of the GPCC is to serve user requirements esp. regarding accuracy of the gridded precipitation analyses and timeliness of the product availability.

As of end-2007 the GPCC database holds world-wide precipitation data of more than 78,000 stations in more than 170 countries featuring more than 19,000,000 station-years of monthly values. Over the past 12 months 40 countries have provided new or updated monthly precipitation time series to GPCC. The GPCC products are adjusted to contribute to WCRP core projects:

  • The GPCC Full data product is adjusted to support Global and regional hydrometeorological research activities in the context of CEOP hydroclimate projects and studies.
  • The GPCC VASClimO product is adjusted to support climate variability and change studies in context of CLIVAR and contributes also to the IPCC Assessment Reports.
  • The GPCC Monitoring product and the supplementary products related to solid/liquid precipitation and systematic error correction are useful to support studies in context of CLIC. In addition GPCC implemented the Arctic Precipitation Data Archive (APDA).

Top of this page

Home


WCRP logo

GEWEX logo


Last Updated:  2010/07/16 12:12 (JST)
This web site is published by support of DIAS
© 2008-2010 CEOP: All Rights Reserved.