IRUS Total

Making State Explicit for Imperative Big Data Processing

File Description SizeFormat 
sdg-atc14-final.pdfPublished version517.4 kBAdobe PDFView/Open
Title: Making State Explicit for Imperative Big Data Processing
Authors: Pietzuch, PR
Fernandez, RC
Migliavacca, M
Kalyvianaki, E
Item Type: Conference Paper
Abstract: Data scientists often implement machine learning algorithms in imperative languages such as Java, Matlab and R. Yet such implementations fail to achieve the performance and scalability of specialised data-parallel processing frameworks. Our goal is to execute imperative Java programs in a data-parallel fashion with high throughput and low latency. This raises two challenges: how to support the arbitrary mutable state of Java programs without compromising scalability, and how to re cover that state after failure with low overhead. Our idea is to infer the dataflow and the types of state accesses from a Java program and use this information to generate a stateful dataflow graph (SDG) . By explicitly separating data from mutablestate, SDGs have specific features to enable this translation: to ensure scalability, distributed state can be partitioned across nodes if computation can occur entirely in parallel; if this is not possible, partial state gives nodes local instances for independent computation, which are reconciled according to application semantics. For fault tolerance, large inmemory state is checkpointed asynchronously without global coordination. We show that the performance of SDGs for several imperative online applications matches that of existing data-parallel processing frameworks.
Issue Date: 1-Jun-2014
Date of Acceptance: 1-Apr-2014
URI: http://hdl.handle.net/10044/1/62212
Publisher: USENIX
Journal / Book Title: Proceedings of USENIX ATC ’14: 2014 USENIX Annual Technical Conference
Copyright Statement: © 2014 The Author(s). This is an open access article.
Sponsor/Funder: Engineering & Physical Science Research Council (EPSRC)
Funder's Grant Number: EP/F035217/1
Conference Name: USENIX Annual Technical Conference (USENIX ATC)
Publication Status: Published
Start Date: 2014-06-19
Finish Date: 2014-06-20
Conference Place: Philadelphia, PA, USA
Online Publication Date: 2014-06-01
Appears in Collections:Faculty of Engineering