TRACK: optimizing artificial neural networks for anomaly detection in Spark Streaming Systems
File(s)VALUETOOLS2020.pdf (747.61 KB)
Published version
Author(s)
Alnafessah, Ahmad
Casale, Giuliano
Type
Conference Paper
Abstract
Due to the growth of Big Data processing technologies and cloudcomputing services, it is common to have multiple tenants share thesame computing resources, which may cause performance anom-alies. There is an urgent need for an effective performance anomalydetection method that can be used within the production environ-ment to avoid any late detection of unexpected system failures.To address this challenge, we introduce, TRACK, a new black-boxtraining workload configurationoptimization with a neural net-work driven methodology to identify anomalous performance inan in-memory Big Data Spark streaming platform. The proposedmethodology revolves around using Bayesian optimization to findthe optimal training dataset size and configuration parameters totrain the model efficiently. TRACK is validated on a real ApacheSpark streaming system and the results show that the TRACKachieves the highest performance (95% for F-score) and reducesthe training time by 80% to efficiently train the proposed anomalydetection model in the in-memory streaming platform.
Date Issued
2020-05-18
Date Acceptance
2020-02-28
Citation
2020, pp.188-191
Publisher
ACM
Start Page
188
End Page
191
Copyright Statement
© 2018 Association for Computing Machinery.
Identifier
https://dl.acm.org/doi/10.1145/3388831.3388860
Source
VALUETOOLS 2020
Subjects
Science & Technology
Technology
Computer Science, Software Engineering
Engineering, Multidisciplinary
Operations Research & Management Science
Telecommunications
Computer Science
Engineering
Performance Anomalies
Apache Spark
Artificial Intelligence
Neural Network
Big Data
Machine Learning
Publication Status
Published
Start Date
2020-05-18
Finish Date
2020-05-20
Coverage Spatial
Tsukuba, Japan
Date Publish Online
2020-05-18