Open and scalable analytics of large Earth observation datasets: from scenes to multidimensional arrays using SciDB and GDAL
File(s)1-s2.0-S0924271617300898-main.pdf (1.89 MB)
Published version
Author(s)
Appel, M
Lahn, F
Buytaert, W
Pebesma, E
Type
Journal Article
Abstract
Earth observation (EO) datasets are commonly provided as collection of scenes, where individual scenes represent a temporal snapshot and cover a particular region on the Earth's surface. Using these data in complex spatiotemporal modeling becomes difficult as soon as data volumes exceed a certain capacity or analyses include many scenes, which may spatially overlap and may have been recorded at different dates. In order to facilitate analytics on large EO datasets, we combine and extend the geospatial data abstraction library (GDAL) and the array-based data management and analytics system SciDB. We present an approach to automatically convert collections of scenes to multidimensional arrays and use SciDB to scale computationally intensive analytics. We evaluate the approach in three study cases on national scale land use change monitoring with Landsat imagery, global empirical orthogonal function analysis of daily precipitation, and combining historical climate model projections with satellite-based observations. Results indicate that the approach can be used to represent various EO datasets and that analyses in SciDB scale well with available computational resources. To simplify analyses of higher-dimensional datasets as from climate model output, however, a generalization of the GDAL data model might be needed. All parts of this work have been implemented as open-source software and we discuss how this may facilitate open and reproducible EO analyses.
Date Issued
2018-02-15
Date Acceptance
2018-01-19
Citation
ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 138, pp.47-56
ISSN
0924-2716
Publisher
Elsevier
Start Page
47
End Page
56
Journal / Book Title
ISPRS Journal of Photogrammetry and Remote Sensing
Volume
138
Copyright Statement
© 2018 The Authors. Published by Elsevier B.V. on behalf of International Society for Photogrammetry and Remote Sensing, Inc. (ISPRS). This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
Sponsor
Natural Environment Research Council (NERC)
Natural Environment Research Council (NERC)
Grant Number
NE/I022558/1
NE/J016578/1
Subjects
0406 Physical Geography And Environmental Geoscience
0909 Geomatic Engineering
Geological & Geomatics Engineering
Publication Status
Published