Abstract
We have witnessed exciting development of RAM technology in the past decade. The memory size grows rapidly and the price continues to decrease, so that it is feasible to deploy large amounts of RAM in a computer system. Several companies and research institutions have devoted a lot of resources to develop in-memory databases (IMDB) that implement queries after loading data into (virtual) memory in advance. The bloom of various in-memory databases pursues us to test and evaluate their performance objectively and fairly. Although the existing database benchmarks like Wisconsin benchmark and TPC-X series have achieved great success, they cannot suit for in-memory databases due to the lack of consideration of unique characteristics of an IMDB. In this study, we propose MemTest, a novel benchmark that concerns some major characteristics of an in-memory database. This benchmark constructs particular metrics, which cover processing time, compression ratio, minimal memory space and column strength of an in-memory database. We design a data model based on inter-bank transaction applications, and a data generator to support uniform and skew data distributions. The MemTest workload includes a set of queries and transactions against the metrics and data model. Finally, we illustrate the efficacy of MemTest through the implementations on two different in-memory databases.
Similar content being viewed by others
References
Gray J. Benchmark handbook: for database and transaction processing systems. San Francisco: Morgan Kaufmann Publishers Inc., 1992
Carey M J, De Witt D J, Naughton J F. The 007 benchmark. In: Buneman P, Jajodia S, eds. SIGMOD Conference. ACM Press, 1993, 12–21
Carey M J, De Witt D J, Naughton J F, Asgarian M, Brown P, Gehrke J E, Shah D N. The buckyobject-relational benchmark (experience paper). In: Proceedings of the 1997 ACM SIGMOD International Conference on Management of Data. 1997, 135–146
Schmidt A, Waas F, Kersten M, Carey M J, Manolescu I, Busse R. Xmark: a benchmark for XML data management. In: Proceedings of the 28th International Conference on Very Large Data Bases. 2002, 974–985
Carey M J, Ling L, Nicola M, Shao L. EXRT: towards a simple benchmark for XML readiness testing. In: Nambiar R, Poess M, eds. Performance Evaluation, Measurement and Characterization of Complex Systems. Springer Berlin Heidelberg, 2010, 93–109
Cooper B F, Silberstein A, Tam E, Ramakrishnan R, Sears R. Benchmarking cloud serving systems with YCSB. In: Proceedings of the 1st ACM Symposium on Cloud Computing. 2010, 143–154
Ghazal A, Rabl T, Hu M, Raab F, Poess M, Crolotte A, Jacobsen H A. BigBench: towards an industry standard benchmark for big data analytics. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data. 2013, 1197–1208
Rabl T, Poess M, Jacobsen H A, O’Neil P, O’ Neil E. Variations of the star schema benchmark to test the effects of data skew on query performance. In: Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering. 2013, 361–372
Färber F, Cha S K, Primsch J, Bornhövd C, Sigg S, Lehner W. SAPHANA database: data management for modern business applications. ACM Sigmod Record, 2012, 40(4): 45–51
Lahiri T, Neimat M A, Folkman S. Oracle TimesTen: an in-memory database for enterprise applications. IEEE Data Engineering Bulletin, 2013, 36(2): 6–13
Diaconu C, Freedman C, Ismert E, Larson P A, Mittal P, Stonecipher R, Verma N, Zwilling M. Hekaton: SQL server’s memory-optimized OLTP engine. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data. 2013, 1243–1254
Gupta M K, Verma V, Verma M S. In-memory database systems — a paradigm shift. International Journal of Engineering Trends and Technology, 2013, 6(6): 333–336
Cole R, Funke F, Giakoumakis L, Guy W, Kemper A, Krompass S, Kuno H, Nambiar R, Neumann T, PoessM, Sattler K U, Seibold M, Simon E, Waas F. The mixed workload CH-benCHmark. In: Proceedings of the 4th ACM International Workshop on Testing Database Systems. 2011
Kang Q, Jin C Q, Zhang Z, Zhou A Y. MemTest: a novel benchmark for in-memory database. In: Zhan J F, Han R, Weng C L, eds. Big Data Benchmarks, Performance Optimization, and Emerging Hardware. Springer International Publishing, 2014, 34–46
Jin C Q, Qian WN, Zhou MQ, Zhou A Y. Benchmarking data management systems: from traditional database to emergent big data. Chinese Journal of Computers, 2015, 38(1): 18–34 (in Chinese)
Cattell R G G, Skeen J. Object operations benchmark. ACM Transactions on Database Systems (TODS), 1992, 17(1): 1–31
Runapongsa K, Patel J M, Jagadish H V, Chen Y, Al-Khalifa S. The Michigan benchmark: towards XML query performance diagnostics. Information Systems, 2006, 31(2): 73–97
Nicola M, Kogan I, Schiefer B. An XML transaction processing benchmark. In: Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data. 2007, 937–948
Werstein P. A performance benchmark for spatiotemporal databases. In: Proceedings of the 10th Annual Colloquium of the Spatial Information Research Centre. 1998, 1365–1374
Düntgen C, Behr T, Güting R H. BerlinMOD: a benchmark for moving object databases. The VLDB Journal, 2009, 18(6): 1335–1368
Arasu A, Cherniack M, Galvez E, Maier D, Maskey A S, Ryvkina E, Ryvkina E, Stonebraker M, Tibbetts R. Linear road: a stream data management benchmark. In: Proceedings of the 30th International Conference on Very Large Data Bases. 2004, 480–491
Tözün P, Pandis I, Kaynak C, Jevdjic D, Ailamaki A. From A to E: analyzing TPC’s OLTP benchmarks: the obsolete, the ubiquitous, the unexplored. In: Proceedings of the 16th ACM International Conference on Extending Database Technology. 2013, 17–28
Liu D W, Luan H, Wang S, Qin B. Main memory database TPC-H workload characterization on modern processor. Journal of Software, 2008, 19(10): 2573–2584
Müller I, Ratsch C, Faerber F. Adaptive string dictionary compression in in-memory column-store database systems. In: Proceedings of the International Conference on Extending Database Technology (EDBT). 2014, 283–294
Abadi D J, Madden S R, Hachem N. Column-storesvs. row-stores: how different are they really? In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. 2008, 967–980
Author information
Authors and Affiliations
Corresponding author
Additional information
Cheqing Jin is a professor at East China Normal University, China. He received his master and bachelor degrees from Zhejiang University, China in 1999 and 2002 respectively, and PhD degree from Fudan University, China in 2005, all in computer science. He worked as an assistant professor in East China University of Science and Technology, China from 2005 to 2008, afterwards he joined ECNU on October 2008. In 2003 and 2007, he visited the HongKong University and the Chinese University of Hongkong respectively. He has acted as the PCmembers for more than ten conferences. His main research interests include streaming data management, location-based services, uncertain data management, data quality, and database benchmarking.
Yangxin Kong received the bachelor degree from Nantong University, China. He is currently working toward the master degree in the Software Engineering Institute at East China Normal University, China. His research interests include technology and application on location based services, data mining, etc.
Qiangqiang Kang received the bachelor and master degrees from Software Engineering Institute at East China Normal University, China. He is now working in China Merchants Bank after graduation. His research interests mainly include database benchmark and reverse query.
Weining Qian is currently a professor in computer science at East China Normal University, China. He received his MS and PhD in computer science from Fudan University, China in 2001 and 2004, respectively. He served as the co-chair of WISE 2012 Challenge, and program committee member of several international conferences, including ICDE 2009/2010/2012 and KDD 2013. His research interests include Web data management and mining of massive data sets.
Aoying Zhou is a professor on computer science at East China Normal University (ECNU), China where he is heading the Institute for Data Science and Engineering. He got his master and bachelor degree in computer science from Sichuan University, China in 1988 and 1985 respectively, and won his PhD degree from Fudan University, China in 1993. Before joining ECNU in 2008, he worked for Fudan University at the Computer Science Department from 1993 to 2007, where he served as the department chair from 1999 to 2002. He worked as a visiting scholar under the Berkeley Scholar Program in UC Berkeley in 2005. He is the winner of the National Science Fund for Distinguished Young Scholars supported by NSFC and the professorship appointment under Changjiang Scholars Program of Ministry of Education. He is now acting as the vice-director of ACM SIGMOD China and Technology Committee on Database of China Computer Federation. He is serving as a member of the editorial boards of some prestigious academic journals, such as VLDB Journal, and WWW Journal. His research interests include Web data management, data management for data-intensive computing, and in-memory data analytics.
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Jin, C., Kong, Y., Kang, Q. et al. Benchmarking in-memory database. Front. Comput. Sci. 10, 1067–1081 (2016). https://doi.org/10.1007/s11704-016-5366-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11704-016-5366-0