research-article

ETune: : Efficient configuration tuning for big-data software systems via configuration space reduction

Authors:

Panpan ZhangsunAuthors Info & Claims

Volume 209, Issue C

https://doi.org/10.1016/j.jss.2023.111936

Published: 14 March 2024 Publication History

Abstract

Configuration tuning for big-data software systems is generally challenging due to the complex configuration space and costly performance evaluation. The workload changes over time in the context of big data making this problem even more difficult. To address the low efficiency issue caused by the need to retune from scratch after workload changes, we present ETune based on a combination of an Bayesian Optimization (BO) based tuner and configuration space reduction techniques to efficiently find high-performance configurations for big-data systems. For configuration space reduction, we develop two approaches: (1) shrinking the configuration space by trading off the exploitation and exploration between the impactful parameters and other parameters in the tuning process, (2) generating the promising regions of configuration space by transferring knowledge across past tuning tasks. The two configuration space reduction methods aim to reduce the huge configuration space to a compact but promising one, and searching in the reduced configuration space can further accelerate the configuration tuning. The extensive experiments show that our configuration space reduction methods considerably boost the BO-based tuner, and ETune significantly improves the tuning efficiency compared with the transfer learning based state-of-the-arts.

Highlights

•

An efficient configuration tuning algorithm for big-data software systems.

•

Improving tuning efficiency through configuration space reduction.

•

The tradeoff between exploitation and exploration in configuration space reduction.

•

Interoperability with other transfer learning-based configuration tuning methods.

References

[1]

Apache Spark Tuning Guide, Apache spark tuning guide, 2019, https://spark.apache.org/docs/latest/tuning.html.

[2]

Apache Storm Performance Tuning, Apache storm performance tuning, 2019, https://storm.apache.org/releases/current/Performance.html.

[3]

Bao L., Liu X., Chen W., Learning-based automatic parameter tuning for big data analytics frameworks, in: 2018 IEEE International Conference on Big Data, Big Data, IEEE, 2018, pp. 181–190.

[4]

Bao L., Liu X., Wang F., Fang B., Actgan: Automatic configuration tuning for software systems with generative adversarial networks, in: 2019 34th IEEE/ACM International Conference on Automated Software Engineering, ASE, IEEE, 2019, pp. 465–476.

[5]

Bao L., Liu X., Xu Z., Fang B., Autoconfig: Automatic configuration tuning for distributed message systems, in: 2018 33rd IEEE/ACM International Conference on Automated Software Engineering, ASE, IEEE, 2018, pp. 29–40.

[6]

Bei Z., Yu Z., Zhang H., Xiong W., Xu C., Eeckhout L., Feng S., RFHOC: A random-forest approach to auto-tuning hadoop’s configuration, IEEE Trans. Parallel Distrib. Syst. 27 (5) (2015) 1470–1483.

[7]

Bergstra J., Bengio Y., Random search for hyper-parameter optimization., J. Mach. Learn. Res. 13 (2) (2012).

[8]

Bindal P.B., Singhal D., Subramanyam A., Kumar V., et al., OneStopTuner: An end to end architecture for JVM tuning of spark applications, 2020, arXiv preprint arXiv:2009.06374.

[9]

Breiman L., Random forests, Mach. Learn. 45 (1) (2001) 5–32.

Digital Library

[10]

Calzarossa M.C., Massari L., Tessera D., Workload characterization: A survey revisited, ACM Comput. Surv. 48 (3) (2016) 1–43.

Digital Library

[11]

Cao, Z., Kuenning, G., Zadok, E., 2020. Carver: Finding important parameters for storage system tuning. In: 18th U S E N I X Conference on File and Storage Technologies (F A S T 20). pp. 43–57.

[12]

Cereda S., Valladares S., Cremonesi P., Doni S., Cgptuner: a contextual Gaussian process bandit approach for the automatic tuning of IT configurations under varying workload conditions, Proc. VLDB Endow. 14 (8) (2021) 1401–1413.

[13]

Chen K., Powers J., Guo S., Tian F., CRESP: Towards optimal resource provisioning for MapReduce computing in public clouds, IEEE Trans. Parallel Distrib. Syst. 25 (6) (2013) 1403–1412.

[14]

Chen J., Xu N., Chen P., Zhang H., Efficient compiler autotuning via bayesian optimization, in: 2021 IEEE/ACM 43rd International Conference on Software Engineering, ICSE, IEEE, 2021, pp. 1198–1209.

[15]

ClouderaSparkTuning J., Cloudera performance management - tuning spark applications, 2018, https://www.cloudera.com/documentation/enterprise/5-9-x/topics/admin_spark_tuning.html.

[16]

ClouderaYarnTuning J., Cloudera performance management - tuning YARN, 2018, https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cdh_ig_yarn_tuning.html.

[17]

Dalibard, V., Schaarschmidt, M., Yoneki, E., 2017. BOAT: Building auto-tuners with structured Bayesian optimization. In: Proceedings of the 26th International Conference on World Wide Web. pp. 479–488.

[18]

Debnath B.K., Lilja D.J., Mokbel M.F., SARD: A statistical approach for ranking database tuning parameters, in: 2008 IEEE 24th International Conference on Data Engineering Workshop, IEEE, 2008, pp. 11–18.

[19]

Dorn, J., Apel, S., Siegmund, N., 2020. Mastering uncertainty in performance estimations of configurable software systems. In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. pp. 684–696.

[20]

Duan S., Thummala V., Babu S., Tuning database configuration parameters with ituned, Proc. VLDB Endow. 2 (1) (2009) 1246–1257.

Digital Library

[21]

Fekry A., Carata L., Pasquier T., Rice A., Accelerating the configuration tuning of big data analytics with similarity-aware multitask Bayesian optimization, in: 2020 IEEE International Conference on Big Data, Big Data, IEEE, 2020, pp. 266–275.

[22]

Fekry, A., Carata, L., Pasquier, T., Rice, A., Hopper, A., 2020. To Tune or Not to Tune? In Search of Optimal Configurations for Data Analytics. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 2494–2504.

[23]

Feurer, M., Letham, B., Bakshy, E., 2018. Scalable meta-learning for bayesian optimization using ranking-weighted gaussian process ensembles. In: AutoML Workshop At ICML, Vol. 7.

[24]

Feurer M., Springenberg J.T., Hutter F., Using meta-learning to initialize Bayesian optimization of hyperparameters, in: MetaSel@ ECAI, Citeseer, 2014, pp. 3–10.

[25]

Fischer L., Gao S., Bernstein A., Machines tuning machines: Configuring distributed stream processors with bayesian optimization, in: 2015 IEEE International Conference on Cluster Computing, IEEE, 2015, pp. 22–31.

[26]

Guo J., Czarnecki K., Apel S., Siegmund N., Wasowski A., Variability-aware performance prediction: A statistical learning approach, in: 2013 28th IEEE/ACM International Conference on Automated Software Engineering, ASE, IEEE, 2013, pp. 301–311.

[27]

Guo J., Yang D., Siegmund N., Apel S., Sarkar A., Valov P., Czarnecki K., Wasowski A., Yu H., Data-efficient performance learning for configurable systems, Empir. Softw. Eng. 23 (3) (2018) 1826–1867.

[28]

Ha H., Zhang H., Deepperf: performance prediction for configurable software with deep sparse neural network, in: 2019 IEEE/ACM 41st International Conference on Software Engineering, ICSE, IEEE, 2019, pp. 1095–1106.

[29]

HadoopTuning H., Hadoop performance tuning tutorial, 2015, http://hadooptutorial.info/hadoop-performance-tuning/.

[30]

HadoopTutorial H., Hadoop MapReduce tutorial, 2018, https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html.

[31]

Helton J.C., Davis F.J., Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems, Reliab. Eng. Syst. Saf. 81 (1) (2003) 23–69.

[32]

Hennig P., Schuler C.J., Entropy search for information-efficient global optimization., J. Mach. Learn. Res. 13 (6) (2012).

[33]

Herodotou H., Babu S., Profiling, what-if analysis, and cost-based optimization of mapreduce programs, Proc. VLDB Endow. 4 (11) (2011) 1111–1122.

Digital Library

[34]

Herodotou H., Lim H., Luo G., Borisov N., Dong L., Cetin F.B., Babu S., Starfish: A self-tuning system for big data analytics., in: Cidr, Vol. 11, No. 2011, 2011, pp. 261–272.

[35]

Hoffman M., Brochu E., De Freitas N., et al., Portfolio allocation for Bayesian optimization, in: UAI, 2011, pp. 327–336.

[36]

Hsu C.-J., Nair V., Freeh V.W., Menzies T., Arrow: Low-level augmented bayesian optimization for finding the best cloud vm, in: 2018 IEEE 38th International Conference on Distributed Computing Systems, ICDCS, IEEE, 2018, pp. 660–670.

[37]

Hsu C.-J., Nair V., Menzies T., Freeh V.W., Scout: An experienced guide to find the best cloud configuration, 2018, arXiv preprint arXiv:1803.01296.

[38]

Huang S., Huang J., Dai J., Xie T., Huang B., The HiBench benchmark suite: Characterization of the MapReduce-based data analysis, in: 2010 IEEE 26th International Conference on Data Engineering Workshops, ICDEW 2010, IEEE, 2010, pp. 41–51.

[39]

Jamshidi P., Casale G., An uncertainty-aware approach to optimal configuration of stream processing systems, in: 2016 IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS, IEEE, 2016, pp. 39–48.

[40]

Jamshidi P., Siegmund N., Velez M., Kästner C., Patel A., Agarwal Y., Transfer learning for performance modeling of configurable systems: An exploratory analysis, in: 2017 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE, IEEE, 2017, pp. 497–508.

[41]

Jones D.R., Schonlau M., Welch W.J., Efficient global optimization of expensive black-box functions, J. Global Optim. 13 (4) (1998) 455–492.

Digital Library

[42]

Kanellis, K., Alagappan, R., Venkataraman, S., 2020. Too many knobs to tune? towards faster database tuning by pre-selecting important knobs. In: 12th U S E N I X Workshop on Hot Topics in Storage and File Systems. HotStorage 20.

[43]

Khan M., Jin Y., Li M., Xiang Y., Jiang C., Hadoop performance modeling for job estimation and resource provisioning, IEEE Trans. Parallel Distrib. Syst. 27 (2) (2015) 441–454.

[44]

Krishna R., Tang C., Sullivan K., Ray B., ConEx: Efficient exploration of big-data system configurations for better performance, IEEE Trans. Softw. Eng. 48 (3) (2020) 893–909.

[45]

Kunjir M., Speeding up AutoTuning of the memory management options in data analytics, Distrib. Parallel Databases 38 (4) (2020) 841–863.

[46]

Lesoil L., Acher M., Blouin A., Jézéquel J.-M., Input sensitivity on the performance of configurable systems an empirical study, J. Syst. Softw. 201 (2023).

[47]

Li, Z.L., Liang, C.-J.M., He, W., Zhu, L., Dai, W., Jiang, J., Sun, G., 2018a. Metis: Robustly tuning tail latencies of cloud systems. In: 2018 U S E N I X Annual Technical Conference (U S E N I X A T C 18). pp. 981–992.

[48]

Li Y., Shen Y., Jiang H., Bai T., Zhang W., Zhang C., Cui B., Transfer learning based search space design for hyperparameter tuning, in: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, Association for Computing Machinery, New York, NY, USA, 2022, pp. 967–977,.

Digital Library

[49]

Li T., Shi S., Luo J., Wang H., A method to identify spark important parameters based on machine learning, in: International Conference of Pioneering Computer Scientists, Engineers and Educators, Springer, 2018, pp. 525–538.

[50]

Lima M.I.V., de Farias V.A., Praciano F.D., Machado J.C., Workload-aware parameter selection and performance prediction for in-memory databases, in: SBBD, 2018, pp. 169–180.

[51]

Liu J., Tang S., Xu G., Ma C., Lin M., A novel configuration tuning method based on feature selection for hadoop MapReduce, IEEE Access 8 (2020) 63862–63871.

[52]

Liu C., Zeng D., Yao H., Hu C., Yan X., Fan Y., Mr-cof: a genetic mapreduce configuration optimization framework, in: International Conference on Algorithms and Architectures for Parallel Processing, Springer, 2015, pp. 344–357.

[53]

Mahgoub, A., Wood, P., Ganesh, S., Mitra, S., Gerlach, W., Harrison, T., Meyer, F., Grama, A., Bagchi, S., Chaterji, S., 2017. Rafiki: A middleware for parameter tuning of nosql datastores for dynamic metagenomics workloads. In: Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference. pp. 28–40.

[54]

Menze B.H., Kelm B.M., Masuch R., Himmelreich U., Bachert P., Petrich W., Hamprecht F.A., A comparison of random forest and its gini importance with standard chemometric methods for the feature selection and classification of spectral data, BMC Bioinformatics 10 (1) (2009) 1–16.

[55]

Mühlbauer S., Sattler F., Kaltenecker C., Dorn J., Apel S., Siegmund N., Analyzing the impact of workloads on modeling the performance of configurable software systems, in: Proceedings of the International Conference on Software Engineering, ICSE, IEEE, 2023.

[56]

Nabi Z., Bouillet E., Bainbridge A., Thomas C., Of streams and storms, Citeseer, 2014, pp. 1–31.

[57]

Nair, V., Menzies, T., Siegmund, N., Apel, S., 2017. Using bad learners to find good configurations. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. pp. 257–267.

[58]

Nair V., Menzies T., Siegmund N., Apel S., Faster discovery of faster system configurations with spectral learning, Autom. Softw. Eng. 25 (2) (2018) 247–277.

Digital Library

[59]

Nair V., Yu Z., Menzies T., Siegmund N., Apel S., Finding faster configurations using flash, IEEE Trans. Softw. Eng. 46 (7) (2018) 794–811.

[60]

Rasmussen C.E., Gaussian processes in machine learning, in: Summer School on Machine Learning, Springer, 2003, pp. 63–71.

[61]

Sampaio, A.R., Beschastnikh, I., Maier, D., Bourne, D., Sundaresen, V., 2023. Auto-tuning elastic applications in production. In: 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice. ICSE-SEIP, pp. 355–367.

[62]

Sarkar A., Guo J., Siegmund N., Apel S., Czarnecki K., Cost-efficient sampling for performance prediction of configurable systems (t), in: 2015 30th IEEE/ACM International Conference on Automated Software Engineering, ASE, IEEE, 2015, pp. 342–352.

[63]

Shahriari B., Swersky K., Wang Z., Adams R.P., De Freitas N., Taking the human out of the loop: A review of Bayesian optimization, Proc. IEEE 104 (1) (2015) 148–175.

[64]

Shi J., Zou J., Lu J., Cao Z., Li S., Wang C., Mrtuner: a toolkit to enable holistic optimization for mapreduce jobs, Proc. VLDB Endow. 7 (13) (2014) 1319–1330.

[65]

Singhal R., Singh P., Performance assurance model for applications on SPARK platform, in: Technology Conference on Performance Evaluation and Benchmarking, Springer, 2017, pp. 131–146.

[66]

Srinivas N., Krause A., Kakade S.M., Seeger M., Gaussian process optimization in the bandit setting: No regret and experimental design, in: Proceedings of the 27th International Conference on Machine Learning, ICML-10, June 21-24, 2010, Haifa, Israel, 2010.

[67]

Sullivan D.G., Seltzer M.I., Pfeffer A., Using probabilistic reasoning to automate software tuning, ACM SIGMETRICS Perform. Eval. Rev. 32 (1) (2004) 404–405.

[68]

Tang, C., 2017. System performance optimization via design and configuration space exploration. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. pp. 1046–1049.

[69]

Thummala, V., Babu, S., 2010. iTuned: a tool for configuring and visualizing database parameters. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data. pp. 1231–1234.

[70]

Trotter M., Liu G., Wood T., Into the storm: Descrying optimal configurations using genetic algorithms and bayesian optimization, in: 2017 IEEE 2nd International Workshops on Foundations and Applications of Self* Systems, FAS* W, IEEE, 2017, pp. 175–180.

[71]

Valov P., Guo J., Czarnecki K., Empirical comparison of regression methods for variability-aware performance prediction, in: Proceedings of the 19th International Conference on Software Product Line, ACM, 2015, pp. 186–190.

[72]

Van Aken, D., Pavlo, A., Gordon, G.J., Zhang, B., 2017. Automatic database management system tuning through large-scale machine learning. In: Proceedings of the 2017 ACM International Conference on Management of Data. pp. 1009–1024.

[73]

Venkataraman, S., Yang, Z., Franklin, M., Recht, B., Stoica, I., 2016. Ernest: Efficient performance prediction for large-scale advanced analytics. In: 13th U S E N I X Symposium on Networked Systems Design and Implementation (N S D I 16). pp. 363–378.

[74]

Verma, A., Cherkasova, L., Campbell, R.H., 2011a. Aria: automatic resource inference and allocation for mapreduce environments. In: Proceedings of the 8th ACM International Conference on Autonomic Computing. pp. 235–244.

[75]

Verma A., Cherkasova L., Campbell R.H., Resource provisioning framework for mapreduce jobs with performance goals, in: ACM/IFIP/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing, Springer, 2011, pp. 165–186.

[76]

Wang K., Khan M.M.H., Performance prediction for apache spark platform, in: 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security, and 2015 IEEE 12th International Conference on Embedded Software and Systems, IEEE, 2015, pp. 166–173.

[77]

Wang K., Lin X., Tang W., Predator—An experience guided configuration optimizer for hadoop MapReduce, in: 4Th IEEE International Conference on Cloud Computing Technology and Science Proceedings, IEEE, 2012, pp. 419–426.

[78]

Wang G., Xu J., He B., A novel method for tuning configuration parameters of spark based on machine learning, in: 2016 IEEE 18th International Conference on High Performance Computing and Communications, IEEE, 2016, pp. 586–593.

[79]

White T., Hadoop: The Definitive Guide, O’Reilly Media, Inc., 2012.

[80]

Wistuba M., Schilling N., Schmidt-Thieme L., Hyperparameter search space pruning–a new component for sequential model-based hyperparameter optimization, in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, 2015, pp. 104–119.

[81]

Yeh C.-C., Zhou J., Chang S.-A., Lin X.-Y., Sun Y., Huang S.-K., Bigexplorer: A configuration recommendation system for big data platform, in: 2016 Conference on Technologies and Applications of Artificial Intelligence, TAAI, IEEE, 2016, pp. 228–234.

[82]

Yu, Z., Bei, Z., Qian, X., 2018. Datasize-aware high dimensional configurations auto-tuning of in-memory cluster computing. In: Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems. pp. 564–577.

[83]

Zacheilas N., Maroulis S., Kalogeraki V., Dione: Profiling spark applications exploiting graph similarity, in: 2017 IEEE International Conference on Big Data, Big Data, IEEE, 2017, pp. 389–394.

[84]

Zacheilas N., Maroulis S., Priovolos T., Kalogeraki V., Gunopulos D., Dione: A framework for automatic profiling and tuning big data applications, in: 2018 IEEE 34th International Conference on Data Engineering, ICDE, IEEE, 2018, pp. 1637–1640.

[85]

Zaouk K., Song F., Lyu C., Sinha A., Diao Y., Shenoy P., Udao: A next-generation unified data analytics optimizer, Proc. VLDB Endow. (PVLDB) 12 (12) (2019).

[86]

Zhang X., Chang Z., Li Y., Wu H., Tan J., Li F., Cui B., Facilitating database tuning with hyper-parameter optimization: a comprehensive experimental evaluation, 2021, arXiv preprint arXiv:2110.12654.

[87]

Zhang B., Van Aken D., Wang J., Dai T., Jiang S., Lao J., Sheng S., Pavlo A., Gordon G.J., A demonstration of the ottertune automatic database management system tuning service, Proc. VLDB Endow. 11 (12) (2018) 1910–1913.

[88]

Zhang, X., Wu, H., Chang, Z., Jin, S., Tan, J., Li, F., Zhang, T., Cui, B., 2021b. Restune: Resource oriented tuning boosted by meta-learning for cloud databases. In: Proceedings of the 2021 International Conference on Management of Data. pp. 2102–2114.

[89]

Zhu Y., Liu J., Classytune: A performance auto-tuner for systems in the cloud, IEEE Trans. Cloud Comput. (2019).

[90]

Zhu, Y., Liu, J., Guo, M., Bao, Y., Ma, W., Liu, Z., Song, K., Yang, Y., 2017. Bestconfig: tapping the performance potential of systems via automatic configuration tuning. In: Proceedings of the 2017 Symposium on Cloud Computing. pp. 338–350.

Recommendations

Experience Transfer for the Configuration Tuning in Large-Scale Computing Systems

This paper proposes a new strategy, the experience transfer, to facilitate the management of large-scale computing systems. It deals with the utilization of management experiences in one system (or previous systems) to benefit the same management task ...
To Tune or Not to Tune?: In Search of Optimal Configurations for Data Analytics
KDD '20: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

This experimental study presents a number of issues that pose a challenge for practical configuration tuning and its deployment in data analytics frameworks. These issues include: 1) the assumption of a static workload or environment, ignoring the ...
KnobTune: A Dynamic Database Configuration Tuning Strategy Leveraging Historical Workload Similarities
CMLDS '24: Proceedings of the International Conference on Computing, Machine Learning and Data Science

Confronted with an abundance of adjustable parameters and ever-shifting workloads, database configuration tuning grapples with persistent challenges. The intricate task of thoroughly optimizing all these configuration "knobs" to attain peak performance ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Journal of Systems and Software

Journal of Systems and Software Volume 209, Issue C

Mar 2024

313 pages

Issue’s Table of Contents

Elsevier Inc.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 14 March 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Oct 2024

Other Metrics

View Author Metrics

Citations

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents