Automating Sensor Data Processing and Q/C with the GCE Data Toolbox

Printer-friendly version

The GCE Data Toolbox is a comprehensive software framework for metadata-based processing, quality control and analysis of environmental data that is provided as open source software by the GCE and CWT LTER programs (https://gce-svn.marsci.uga.edu/trac/GCE_Toolbox/). The toolbox is an add-on library for the MATLAB technical computing language, providing an EML-compatible data and metadata model and suite of functions for both command-line and GUI-driven environmental data management. LTER Network ARRA funding was provided in 2012-2013 to enhance the usability and documentation of this software, and this effort along with related sensor data management training at LNO have resulted in significant adoption at LTER sites. This software is recommended by the ESIP EnviroSensing Cluster (http://wiki.esipfed.org/index.php/EnviroSensing_Cluster) as a middleware framework for managing real-time sensor data, and it has been successfully integrated with other environmental data management software including Data Turbine, the CUAHSI Observations Data Model and Kepler. In addition, this toolbox can both parse and generate PASTA-compliant Ecological Metadata Language (EML) metadata and data packages, making it an effective workflow engine for both publishing and synthesizing data in the LTER Network Data Catalog.

Initial implementation of the GCE Data Toolbox to automate sensor data processing has been successful at many LTER sites, however significant challenges have been encountered in two primary areas: 1) developing effective Q/C rules for automated data flagging; 2) developing and managing custom workflows for site-specific sensor configurations and environmental conditions. This working group session will attempt to address both of those problem areas through a combination of live demonstrations, directed training exercises and peer-based learning. Current site practices and input from group discussions will be documented and used to draft a set of best practices and corresponding use cases for contribution to both the GCE Data Toolbox and ESIP EnviroSensing Wikis, and included in online training materials for the software. This input will also help direct future development of this community-oriented software, and potentially contribute to an article in preparation for a special issue of Ecological Informatics on long-term ecological data management.
 

Working Group Reports

Final reports: 

Working Group Report - Automating Sensor Data Processing and Q/C with the GCE Data Toolbox

Participant list: 

Attendees_Web.xlsx