SCABBARD: single-node fault-tolerant stream processing
File(s)p361-theodorakis.pdf (594.52 KB)
Published version
Author(s)
Theodorakis, Georgios
Kounelis, Fotios
Pietzuch, Peter
Pirk, Holger
Type
Conference Paper
Abstract
Single-node multi-core stream processing engines (SPEs) can process hundreds of millions of tuples per second. Yet making them fault-tolerant with exactly-once semantics while retaining this performance is an open challenge: due to the limited I/O bandwidth of a single-node, it becomes infeasible to persist all stream data and operator state during execution. Instead, single-node SPEs rely on upstream distributed systems, such as Apache Kafka, to recover stream data after failure, necessitating complex cluster-based deployments. This lack of built-in fault-tolerance features has hindered the adoption of single-node SPEs.We describe Scabbard, the first single-node SPE that supports exactly-once fault-tolerance semantics despite limited local I/O bandwidth. Scabbard achieves this by integrating persistence operations with the query workload. Within the operator graph, Scabbard determines when to persist streams based on the selectivity of operators: by persisting streams after operators that discard data, it can substantially reduce the required I/O bandwidth. As part of the operator graph, Scabbard supports parallel persistence operations and uses markers to decide when to discard persisted data. The persisted data volume is further reduced using workload-specific compression: Scabbard monitors stream statistics and dynamically generates computationally efficient compression operators. Our experiments show that Scabbard can execute stream queries that process over 200 million tuples per second while recovering from failures with sub-second latencies.
Date Issued
2022-09-05
Date Acceptance
2022-09-01
Citation
Proceedings of the VLDB Endowment, 2022, 15 (2), pp.361-374
ISSN
2150-8097
Publisher
VLDB Endowment
Start Page
361
End Page
374
Journal / Book Title
Proceedings of the VLDB Endowment
Volume
15
Issue
2
Copyright Statement
© 2022 The Author(s). This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by
emailing info@vldb.org. Copyright is held by the owner/author(s). Publication rights
licensed to the VLDB Endowment
emailing info@vldb.org. Copyright is held by the owner/author(s). Publication rights
licensed to the VLDB Endowment
Identifier
https://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000757032100019&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=a2bf6146997ec60c407a63945d4e92bb
Source
48th International Conference on Very Large Data Bases (VLDB)
Subjects
Computer Science
Computer Science, Information Systems
Computer Science, Theory & Methods
RECOVERY
Science & Technology
STATE MANAGEMENT
Technology
TIME
Publication Status
Published
Start Date
2022-09-05
Finish Date
2022-09-09
Coverage Spatial
Sydney, Australia