Open and scalable analytics of large Earth observation datasets: from scenes to multidimensional arrays using SciDB and GDAL

File Description SizeFormat 
1-s2.0-S0924271617300898-main.pdfPublished version1.94 MBAdobe PDFView/Open
Title: Open and scalable analytics of large Earth observation datasets: from scenes to multidimensional arrays using SciDB and GDAL
Authors: Appel, M
Lahn, F
Buytaert, W
Pebesma, E
Item Type: Journal Article
Abstract: Earth observation (EO) datasets are commonly provided as collection of scenes, where individual scenes represent a temporal snapshot and cover a particular region on the Earth's surface. Using these data in complex spatiotemporal modeling becomes difficult as soon as data volumes exceed a certain capacity or analyses include many scenes, which may spatially overlap and may have been recorded at different dates. In order to facilitate analytics on large EO datasets, we combine and extend the geospatial data abstraction library (GDAL) and the array-based data management and analytics system SciDB. We present an approach to automatically convert collections of scenes to multidimensional arrays and use SciDB to scale computationally intensive analytics. We evaluate the approach in three study cases on national scale land use change monitoring with Landsat imagery, global empirical orthogonal function analysis of daily precipitation, and combining historical climate model projections with satellite-based observations. Results indicate that the approach can be used to represent various EO datasets and that analyses in SciDB scale well with available computational resources. To simplify analyses of higher-dimensional datasets as from climate model output, however, a generalization of the GDAL data model might be needed. All parts of this work have been implemented as open-source software and we discuss how this may facilitate open and reproducible EO analyses.
Issue Date: 15-Feb-2018
Date of Acceptance: 19-Jan-2018
ISSN: 0924-2716
Publisher: Elsevier
Start Page: 47
End Page: 56
Journal / Book Title: ISPRS Journal of Photogrammetry and Remote Sensing
Volume: 138
Copyright Statement: © 2018 The Authors. Published by Elsevier B.V. on behalf of International Society for Photogrammetry and Remote Sensing, Inc. (ISPRS). This is an open access article under the CC BY license (
Sponsor/Funder: Natural Environment Research Council (NERC)
Natural Environment Research Council (NERC)
Funder's Grant Number: NE/I022558/1
Keywords: 0406 Physical Geography And Environmental Geoscience
0909 Geomatic Engineering
Geological & Geomatics Engineering
Publication Status: Published
Open Access location:
Appears in Collections:Civil and Environmental Engineering
Centre for Environmental Policy
Faculty of Natural Sciences

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

Creative Commons