research-article

Open access

To Tune or Not to Tune?: In Search of Optimal Configurations for Data Analytics

Authors:

Thomas Pasquier,

Andy HopperAuthors Info & Claims

KDD '20: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Pages 2494 - 2504

https://doi.org/10.1145/3394486.3403299

Published: 20 August 2020 Publication History

Abstract

This experimental study presents a number of issues that pose a challenge for practical configuration tuning and its deployment in data analytics frameworks. These issues include: 1) the assumption of a static workload or environment, ignoring the dynamic characteristics of the analytics environment (e.g., increase in input data size, changes in allocation of resources). 2) the amortization of tuning costs and how this influences what workloads can be tuned in practice in a cost-effective manner. 3) the need for a comprehensive incremental tuning solution for a diverse set of workloads. We adapt different ML techniques in order to obtain efficient incremental tuning in our problem domain, and propose Tuneful, a configuration tuning framework. We show how it is designed to overcome the above issues and illustrate its applicability by running a wide array of experiments in cloud environments provided by two different service providers.

References

[1]

TPC-H SQL benchmark, 2014. http://www.tpc.org/tpch/.

[2]

Apache Spark: fast and general engine for large-scale data processing, 2015. https://spark.apache.org/.

[3]

Amazon EC2 instance Pricing, 2018. https://aws.amazon.com/ec2/pricing/on-demand/.

[4]

Hadoop distributed file system, 2018. https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html.

[5]

Tuneful: Experiment data repository, 2020. https://github.com/ayat-khairy/tuneful-data.git.

[6]

Tuneful: project repository, 2020. https://github.com/ayat-khairy/tuneful-code.git.

[7]

Omid Alipourfard, Hongqiang Harry Liu, Jianshu Chen, Shivaram Venkataraman, Minlan Yu, and Ming Zhang. Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics. In NSDI, volume 2, pages 4--2, 2017.

Digital Library

[8]

Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni, Jonathan Ragan-Kelley, Jeffrey Bosboom, Una-May O'Reilly, and Saman Amarasinghe. Opentuner: An extensible framework for program autotuning. In Parallel Architecture and Compilation Techniques (PACT), 2014 23rd International Conference on, pages 303--315. IEEE, 2014.

[9]

Emanuele Borgonovo and Elmar Plischke. Sensitivity analysis: a review of recent advances. European Journal of Operational Research, 248(3):869--887, 2016.

[10]

Marc Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data-efficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):408--423, 2015.

Digital Library

[11]

Songyun Duan, Vamsidhar Thummala, and Shivnath Babu. Tuning database configuration parameters with ituned. Proceedings of the VLDB Endowment, 2(1):1246--1257, 2009.

Digital Library

[12]

Ayat Fekry, Lucian Carata, Thomas Pasquier, Andrew Rice, and Andy Hopper. Tuneful: An online significance-aware configuration tuner for big data analytics, 2020. https://arxiv.org/pdf/2001.08002.pdf.

[13]

Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. Gene selection for cancer classification using support vector machines. Machine learning, 46(1--3):389--422, 2002.

[14]

Shengsheng Huang, Jie Huang, Jinquan Dai, Tao Xie, and Bo Huang. The HiBench benchmark suite: Characterization of the MapReduce-based data analysis. In Data Engineering Workshops (ICDEW), 2010 IEEE 26th International Conference on, pages 41--51. IEEE, 2010.

[15]

SPT Krishnan and Jose L Ugia Gonzalez. Google compute engine. In Building Your Next Big Thing with Google Cloud Platform, pages 53--81. Springer, 2015.

[16]

Palden Lama and Xiaobo Zhou. Aroma: Automated resource allocation and configuration of MapReduce environment in the cloud. In Proceedings of the 9th international conference on Autonomic computing, pages 63--72. ACM, 2012.

Digital Library

[17]

Guangdeng Liao, Kushal Datta, and Theodore L Willke. Gunther: Search-based auto-tuning of MapReduce. In European Conference on Parallel Processing, pages 406--419. Springer, 2013.

Digital Library

[18]

Andy Liaw, Matthew Wiener, et al. Classification and regression by randomforest. R news, 2(3):18--22, 2002.

[19]

Ashraf Mahgoub, Paul Wood, Sachandhan Ganesh, Subrata Mitra, Wolfgang Gerlach, Travis Harrison, Folker Meyer, Ananth Grama, Saurabh Bagchi, and Somali Chaterji. Rafiki: A middleware for parameter tuning of NoSQL datastores for dynamic metagenomics workloads. In Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference, pages 28--40. ACM, 2017.

Digital Library

[20]

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148--175, 2015.

[21]

Ilya M Sobol. On quasi-monte carlo integrations. Mathematics and computers in simulation, 47(2--5):103--112, 1998.

[22]

Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. In Advances in neural information processing systems, pages 2004--2012, 2013.

Digital Library

[23]

Dana Van Aken, Andrew Pavlo, Geoffrey J Gordon, and Bohan Zhang. Automatic database management system tuning through large-scale machine learning. In Proceedings of the 2017 ACM International Conference on Management of Data, pages 1009--1024. ACM, 2017.

Digital Library

[24]

Guolu Wang, Jungang Xu, and Ben He. A novel method for tuning configuration parameters of Spark based on machine learning. In High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), 2016 IEEE 18th International Conference on, pages 586--593. IEEE, 2016.

[25]

John A Weymark. Generalized Gini inequality indices. Mathematical Social Sciences, 1(4):409--430, 1981.

[26]

Zhibin Yu, Zhendong Bei, and Xuehai Qian. Datasize-aware high dimensional configurations auto-tuning of in-memory cluster computing. In Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems, pages 564--577. ACM, 2018.

Digital Library

[27]

Yuqing Zhu, Jianxun Liu, Mengying Guo, Yungang Bao, Wenlong Ma, Zhuoyue Liu, Kunpeng Song, and Yingchun Yang. Bestconfig: tapping the performance potential of systems via automatic configuration tuning. In Proceedings of the 2017 Symposium on Cloud Computing, pages 338--350. ACM, 2017.

Digital Library

Cited By

Zhang XWu HLi YTang ZTan JLi FCui B(2024)An Efficient Transfer Learning Based Configuration Adviser for Database TuningProceedings of the VLDB Endowment10.14778/3632093.363211417:3(539-552)Online publication date: 20-Jan-2024
https://doi.org/10.14778/3632093.3632114
Mu TWang HTang HShao X(2024)ShrinkHPO: Towards Explainable Parallel Hyperparameter Optimization2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00371(4897-4910)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00371
Jang DYoon HJung KChung Y(2024) QHB + : Accelerated Configuration Optimization for Automated Performance Tuning of Spark SQL Applications IEEE Access10.1109/ACCESS.2024.339133312(60138-60148)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3391333
Show More Cited By

Index Terms

To Tune or Not to Tune?: In Search of Optimal Configurations for Data Analytics
1. Theory of computation
  1. Design and analysis of algorithms
    1. Mathematical optimization
      1. Non-parametric optimization
    2. Online algorithms
      1. Online learning algorithms
  2. Theory and algorithms for application domains
    1. Machine learning theory
      1. Kernel methods
        Gaussian processes

Recommendations

ETune: Efficient configuration tuning for big-data software systems via configuration space reduction
Abstract
Configuration tuning for big-data software systems is generally challenging due to the complex configuration space and costly performance evaluation. The workload changes over time in the context of big data making this problem even more ...
Highlights
- An efficient configuration tuning algorithm for big-data software systems.
- Improving tuning efficiency through configuration space reduction.
- The tradeoff between exploitation and exploration in configuration space reduction.
- ...
KnobTune: A Dynamic Database Configuration Tuning Strategy Leveraging Historical Workload Similarities
CMLDS '24: Proceedings of the International Conference on Computing, Machine Learning and Data Science

Confronted with an abundance of adjustable parameters and ever-shifting workloads, database configuration tuning grapples with persistent challenges. The intricate task of thoroughly optimizing all these configuration "knobs" to attain peak performance ...
Experience Transfer for the Configuration Tuning in Large-Scale Computing Systems

This paper proposes a new strategy, the experience transfer, to facilitate the management of large-scale computing systems. It deals with the utilization of management experiences in one system (or previous systems) to benefit the same management task ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '20: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

August 2020

3664 pages

ISBN:9781450379984

DOI:10.1145/3394486

General Chairs:
Rajesh Gupta
UC San Diego, USA
,
Yan Liu
USC, USA
,
Program Chairs:
Mohak Shah
LG Electronics, USA
,
Suju Rajan
Linkedin, USA
,
Publications Chairs:
Jiliang Tang
Michigan State, USA
,
B. Aditya Prakash
Georgia Tech, USA

Copyright © 2020 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 August 2020

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '20

Sponsor:

KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

July 6 - 10, 2020

CA, Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '24

Sponsor:
sigkdd
sigkdd

The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 25 - 29, 2024

Barcelona , Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
1,078
Total Downloads

Downloads (Last 12 months)178
Downloads (Last 6 weeks)25

Reflects downloads up to 14 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang XWu HLi YTang ZTan JLi FCui B(2024)An Efficient Transfer Learning Based Configuration Adviser for Database TuningProceedings of the VLDB Endowment10.14778/3632093.363211417:3(539-552)Online publication date: 20-Jan-2024
https://doi.org/10.14778/3632093.3632114
Mu TWang HTang HShao X(2024)ShrinkHPO: Towards Explainable Parallel Hyperparameter Optimization2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00371(4897-4910)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00371
Jang DYoon HJung KChung Y(2024) QHB + : Accelerated Configuration Optimization for Automated Performance Tuning of Spark SQL Applications IEEE Access10.1109/ACCESS.2024.339133312(60138-60148)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3391333
Li ZTu YMa Z(2023)A Sample-Aware Database Tuning System With Deep Reinforcement LearningJournal of Database Management10.4018/JDM.33351935:1(1-25)Online publication date: 9-Nov-2023
https://doi.org/10.4018/JDM.333519
Shi LLi TWei LTao YLi CGao Y(2023)FASTune: Towards Fast and Stable Database Tuning System with Reinforcement LearningElectronics10.3390/electronics1210216812:10(2168)Online publication date: 10-May-2023
https://doi.org/10.3390/electronics12102168
Li YJiang HShen YFang YYang XHuang DZhang XZhang WZhang CChen PCui B(2023)Towards General and Efficient Online Tuning for SparkProceedings of the VLDB Endowment10.14778/3611540.361154816:12(3570-3583)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.14778/3611540.3611548
Dou HWei XWang KZhang YChen PHuang Y(2023)EFTuner: A Bi-Objective Configuration Parameter Auto-Tuning Method Towards Energy-Efficient Big Data ProcessingProceedings of the 14th Asia-Pacific Symposium on Internetware10.1145/3609437.3609443(292-301)Online publication date: 4-Aug-2023
https://dl.acm.org/doi/10.1145/3609437.3609443
Zhang XChang ZWu HLi YChen JTan JLi FCui B(2023)A Unified and Efficient Coordinating Framework for Autonomous DBMS TuningProceedings of the ACM on Management of Data10.1145/35893311:2(1-26)Online publication date: 20-Jun-2023
https://doi.org/10.1145/3589331
Huang SWang ZZhang XTu YLi ZCui B(2023)DBPA: A Benchmark for Transactional Database Performance AnomaliesProceedings of the ACM on Management of Data10.1145/35889261:1(1-26)Online publication date: 30-May-2023
https://doi.org/10.1145/3588926
Scheinert DWiesner PWittkopp TThamsen LWill JKao O(2023)Karasu: A Collaborative Approach to Efficient Cluster Configuration for Big Data Analytics2023 IEEE International Performance, Computing, and Communications Conference (IPCCC)10.1109/IPCCC59175.2023.10253884(403-412)Online publication date: 17-Nov-2023
https://doi.org/10.1109/IPCCC59175.2023.10253884
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents