AT-GIS: highly parallel spatial query processing with associative transducers
Author(s)
Ogden
Thomas, D
Pietzuch, P
Type
Conference Paper
Abstract
Users in many domains, including urban planning, transportation,
and environmental science want to execute analytical queries over
continuously updated spatial datasets. Current solutions for largescale
spatial query processing either rely on extensions to RDBMS,
which entails expensive loading and indexing phases when the
data changes, or distributed map/reduce frameworks, running on
resource-hungry compute clusters. Both solutions struggle with the
sequential bottleneck of parsing complex, hierarchical spatial data
formats, which frequently dominates query execution time. Our
goal is to fully exploit the parallelism offered by modern multicore
CPUs for parsing and query execution, thus providing the
performance of a cluster with the resources of a single machine.
We describe AT-GIS, a highly-parallel spatial query processing
system that scales linearly to a large number of CPU cores. ATGIS
integrates the parsing and querying of spatial data using a new
computational abstraction called associative transducers(ATs). ATs
can form a single data-parallel pipeline for computation without
requiring the spatial input data to be split into logically independent
blocks. Using ATs, AT-GIS can execute, in parallel, spatial query
operators on the raw input data in multiple formats, without any
pre-processing. On a single 64-core machine, AT-GIS provides 3×
the performance of an 8-node Hadoop cluster with 192 cores for
containment queries, and 10× for aggregation queries.
and environmental science want to execute analytical queries over
continuously updated spatial datasets. Current solutions for largescale
spatial query processing either rely on extensions to RDBMS,
which entails expensive loading and indexing phases when the
data changes, or distributed map/reduce frameworks, running on
resource-hungry compute clusters. Both solutions struggle with the
sequential bottleneck of parsing complex, hierarchical spatial data
formats, which frequently dominates query execution time. Our
goal is to fully exploit the parallelism offered by modern multicore
CPUs for parsing and query execution, thus providing the
performance of a cluster with the resources of a single machine.
We describe AT-GIS, a highly-parallel spatial query processing
system that scales linearly to a large number of CPU cores. ATGIS
integrates the parsing and querying of spatial data using a new
computational abstraction called associative transducers(ATs). ATs
can form a single data-parallel pipeline for computation without
requiring the spatial input data to be split into logically independent
blocks. Using ATs, AT-GIS can execute, in parallel, spatial query
operators on the raw input data in multiple formats, without any
pre-processing. On a single 64-core machine, AT-GIS provides 3×
the performance of an 8-node Hadoop cluster with 192 cores for
containment queries, and 10× for aggregation queries.
Date Issued
2016-06
Date Acceptance
2015-11-11
Citation
2016, pp.1041-1054
Publisher
ACM
Start Page
1041
End Page
1054
Replaces
10044/1/30265
Copyright Statement
© 2016 ACM
Identifier
https://dl.acm.org/doi/10.1145/2882903.2882962
Source
ACM SIGMOD International Conference on Management of Data 2016
Subjects
Science & Technology
Technology
Computer Science, Information Systems
Computer Science
Publication Status
Published
Start Date
2016-06-26
Finish Date
2016-07-01
Coverage Spatial
San Francisco, USA
Date Publish Online
2016-06