Experiments or simulation? A characterization of evaluation methods for in-memory databases
File(s)camera_ready_pdfexpress.pdf (671.08 KB)
Accepted version
Author(s)
Molka, K
Casale, G
Type
Conference Paper
Abstract
The recent growth of interest for in-memory databases poses the question on whether established prediction methods such as response surfaces and simulation are effective to describe the performance of these systems. In particular, the limited dependence of in-memory technologies on the disk makes methods such as simulation more appealing than in the past, since disks are difficult to simulate. To answer this question, we study an in-memory commercial solution, SAP HANA, deployed on a high-end server with 120 physical cores. First, we apply experimental design methods to generate response surfaces that describe database performance as a function of workload and hardware parameters. Next, we develop a class-switching queueing network model to predict in-memory database performance under similar scenarios. By comparing the applicability of the two approaches to modeling multi-tenancy, we find that both queueing and response surface models yield mean prediction errors in the range 5%-22% with respect to mean memory occupancy and response times, but the accuracy for the latter deteriorates in response surfaces as the number of experiments are reduced, whereas simulation is effective in all cases. This suggests that simulation can be very effective in performance prediction for in-memory database management.
Date Issued
2015-11-09
Date Acceptance
2015-08-18
Citation
Network and Service Management (CNSM), 2015 11th International Conference on, 2015, pp.201-209
ISBN
978-3-9018-8278-4
Publisher
IEEE
Start Page
201
End Page
209
Journal / Book Title
Network and Service Management (CNSM), 2015 11th International Conference on
Copyright Statement
© 2015 IFIP
Sponsor
Commission of the European Communities
Grant Number
644869
Source
IEEE CNSM 2015
Publication Status
Published
Start Date
2015-11-09
Finish Date
2015-11-13
Coverage Spatial
Barcelona