research-article

ACTGAN: automatic configuration tuning for software systems with generative adversarial networks

Authors:

Fangzheng Wang,

Baoyin FangAuthors Info & Claims

ASE '19: Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering

Pages 465 - 476

https://doi.org/10.1109/ASE.2019.00051

Published: 07 February 2020 Publication History

Abstract

Complex software systems often provide a large number of parameters so that users can configure them for their specific application scenarios. However, configuration tuning requires a deep understanding of the software system, far beyond the abilities of typical system users. To address this issue, many existing approaches focus on exploring and learning good performance estimation models. The accuracy of such models often suffers when the number of available samples is small, a thorny challenge under a given tuning-time constraint. By contrast, we hypothesize that good configurations often share certain hidden structures. Therefore, instead of trying to improve the performance estimation of a given configuration, we focus on capturing the hidden structures of good configurations and utilizing such learned structure to generate potentially better configurations. We propose ACTGAN to achieve this goal. We have implemented and evaluated ACTGAN using 17 workloads with eight different software systems. Experimental results show that ACTGAN outperforms default configurations by 76.22% on average, and six state-of-the-art configuration tuning algorithms by 6.58%-64.56%. Furthermore, the ACTGAN-generated configurations are often better than those used in training and show certain features consisting with domain knowledge, both of which supports our hypothesis.

References

[1]

spark, "http://spark.apache.org," Accessed on Decemeber 20th 2018.

[2]

HBase, "https://hbase.apache.org/," Accessed on July 31th 2018.

[3]

H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong, F. B. Cetin, and S. Babu, "Starfish: A self-tuning system for big data analytics." in CIDR, vol. 11, 2011, pp. 261--272.

[4]

Y. Zhu, J. Liu, M. Guo, Y. Bao, W. Ma, Z. Liu, K. Song, and Y. Yang, "Bestconfig: tapping the performance potential of systems via automatic configuration tuning," in Proceedings of the 2017 Symposium on Cloud Computing. ACM, 2017, pp. 338--350.

[5]

Z. Bei, Z. Yu, H. Zhang, W. Xiong, C. Xu, L. Eeckhout, and S. Feng, "Rfhoc: A random-forest approach to auto-tuning hadoop's configuration," IEEE Transactions on Parallel and Distributed Systems, vol. 27, no. 5, pp. 1470--1483, 2016.

Digital Library

[6]

J. Guo, D. Yang, N. Siegmund, S. Apel, A. Sarkar, P. Valov, K. Czarnecki, A. Wasowski, and H. Yu, "Data-efficient performance learning for configurable systems," Empirical Software Engineering, vol. 23, no. 3, pp. 1826--1867, 2018.

Digital Library

[7]

C. Liu, D. Zeng, H. Yao, C. Hu, X. Yan, and Y. Fan, "Mr-cof: a genetic mapreduce configuration optimization framework," in International Conference on Algorithms and Architectures for Parallel Processing. Springer, 2015, pp. 344--357.

[8]

K. Wang and M. M. H. Khan, "Performance prediction for apache spark platform," in High Performance Computing and Communications (HPCC), 2015 IEEE 17th International Conference on. IEEE, 2015, pp. 166--173.

[9]

T. Xu, L. Jin, X. Fan, Y. Zhou, S. Pasupathy, and R. Talwadker, "Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software," in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 2015, pp. 307--319.

[10]

Kafka, "https://kafka.apache.org/," Accessed on July 31th 2018.

[11]

Hive, "https://hive.apache.org/," Accessed on July 31th 2018.

[12]

Redis, "https://redis.io/," Accessed on July 31th 2018.

[13]

MySQL, "https://www.mysql.com/," Accessed on July 31th 2018.

[14]

Cassandra, "http://cassandra.apache.org/," Accessed on July 31th 2018.

[15]

Tomcat, "http://tomcat.apache.org/," Accessed on July 31th 2018.

[16]

T. Ye and S. Kalyanaraman, "A recursive random search algorithm for large-scale network parameter configuration," ACM SIGMETRICS Performance Evaluation Review, vol. 31, no. 1, pp. 196--205, 2003.

Digital Library

[17]

M. Woodside, G. Franks, and D. C. Petriu, "The future of software performance engineering," in 2007 Future of Software Engineering. IEEE Computer Society, 2007, pp. 171--187.

[18]

S. Balsamo, A. Di Marco, P. Inverardi, and M. Simeoni, "Model-based performance prediction in software development: A survey," IEEE Transactions on Software Engineering, vol. 30, no. 5, pp. 295--310, 2004.

Digital Library

[19]

H. Koziolek, "Performance evaluation of component-based software systems: A survey," Performance Evaluation, vol. 67, no. 8, pp. 634--658, 2010.

Digital Library

[20]

D. Paluch, H. Kienegger, and H. Krcmar, "A workload-dependent performance analysis of an in-memory database in a multi-tenant configuration," in Companion of the 2018 ACM/SPEC International Conference on Performance Engineering. ACM, 2018, pp. 131--134.

[21]

B. Chen, Y. Liu, and W. Le, "Generating performance distributions via probabilistic symbolic execution," in Proceedings of the 38th International Conference on Software Engineering. ACM, 2016, pp. 49--60.

[22]

A. Abdelaziz, W. Kadir, and A. Osman, "Comparative analysis of software performance prediction approaches in context of component-based system," International Journal of Computer Applications, vol. 23, no. 3, pp. 15--22, 2011.

[23]

J. Bergstra and Y. Bengio, "Random search for hyper-parameter optimization," Journal of Machine Learning Research, vol. 13, no. Feb, pp. 281--305, 2012.

Digital Library

[24]

T. Wang, M. Harman, Y. Jia, and J. Krinke, "Searching for better configurations: a rigorous approach to clone evaluation," in Proceedings of the 9th Joint Meeting on Foundations of Software Engineering. ACM, 2013, pp. 455--465.

[25]

V. Nair, T. Menzies, N. Siegmund, and S. Apel, "Using bad learners to find good configurations," in Proceedings of the 11th Joint Meeting on Foundations of Software Engineering. ACM, 2017, pp. 257--267.

[26]

J. Oh, D. Batory, M. Myers, and N. Siegmund, "Finding near-optimal configurations in product lines by random sampling," in Proceedings of the 11th Joint Meeting on Foundations of Software Engineering. ACM, 2017, pp. 61--71.

[27]

R. Olaechea, D. Rayside, J. Guo, and K. Czarnecki, "Comparison of exact and approximate multi-objective optimization for software product lines," in Proceedings of the 18th International Software Product Line Conference-Volume 1. ACM, 2014, pp. 92--101.

[28]

F. Wu, W. Weimer, M. Harman, Y. Jia, and J. Krinke, "Deep parameter optimisation," in Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation. ACM, 2015, pp. 1375--1382.

[29]

B. Xi, Z. Liu, M. Raghavachari, C. H. Xia, and L. Zhang, "A smart hill-climbing algorithm for application server configuration," in Proceedings of the 13th international conference on World Wide Web. ACM, 2004, pp. 287--296.

[30]

J. Snoek, H. Larochelle, and R. P. Adams, "Practical bayesian optimization of machine learning algorithms," in Advances in neural information processing systems, 2012, pp. 2951--2959.

[31]

A. Sarkar, J. Guo, N. Siegmund, S. Apel, and K. Czarnecki, "Cost-efficient sampling for performance prediction of configurable systems," in Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on. IEEE, 2015, pp. 342--352.

[32]

N. B. Rizvandi, J. Taheri, R. Moraveji, and A. Y. Zomaya, "On modelling and prediction of total cpu usage for applications in mapreduce environments," in International Conference on Algorithms and Architectures for Parallel Processing. Springer, 2012, pp. 414--427.

[33]

M. Niu, S. Rogers, M. Filippone, and D. Husmeier, "Fast inference in nonlinear dynamical systems using gradient matching," in Journal of Machine Learning Research: Workshop and Conference Proceedings, vol. 48. Journal of Machine Learning Research, 2016, pp. 1699--1707.

[34]

S. Wang, C. Li, H. Hoffmann, S. Lu, W. Sentosa, and A. I. Kistijantoro, "Understanding and auto-adjusting performance-sensitive configurations," in ACM SIGPLAN Notices, vol. 53, no. 2. ACM, 2018, pp. 154--168.

[35]

P. Lama and X. Zhou, "Aroma: Automated resource allocation and configuration of mapreduce environment in the cloud," in Proceedings of the 9th international conference on Autonomic computing. ACM, 2012, pp. 63--72.

[36]

N. Luo, Z. Yu, Z. Bei, C. Xu, C. Jiang, and L. Lin, "Performance modeling for spark using svm," in Cloud Computing and Big Data (CCBD), 2016 7th International Conference on. IEEE, 2016, pp. 127--131.

[37]

Y. Zhang, J. Guo, E. Blais, and K. Czarnecki, "Performance prediction of configurable software systems by fourier learning," in Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on. IEEE, 2015, pp. 365--373.

Digital Library

[38]

C. Tang, "System performance optimization via design and configuration space exploration," in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 2017, pp. 1046--1049.

[39]

S. Soltani, M. Asadi, M. Hatala, D. Gašević, and E. Bagheri, "Automated planning for feature model configuration based on stakeholders' business concerns," in Automated Software Engineering (ASE), 2011 26th IEEE/ACM International Conference on. IEEE, 2011, pp. 536--539.

[40]

V. Nair, T. Menzies, N. Siegmund, and S. Apel, "Faster discovery of faster system configurations with spectral learning," Automated Software Engineering, pp. 1--31, 2017.

[41]

C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto, "Automated parameter optimization of classification techniques for defect prediction models," in Software Engineering (ICSE), 2016 IEEE/ACM 38th International Conference on. IEEE, 2016, pp. 321--332.

[42]

S. Falkner, A. Klein, and F. Hutter, "BOHB: Robust and efficient hyperparameter optimization at scale," in Proceedings of the 35th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. Stockholmsmssan, Stockholm Sweden: PMLR, 10--15 Jul 2018, pp. 1436--1445. [Online]. Available: http://proceedings.mlr.press/v80/falkner18a.html

[43]

F. Hutter, H. Hoos, and K. L-Brown, "Sequential model based optimization for general algorithm configuration." LION, vol. 5, pp. 507--523, 2011.

[44]

J. Bergstra, D. Yamins, and D. D. Cox, "Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms," in Proceedings of the 12th Python in Science Conference. Citeseer, 2013, pp. 13--20.

[45]

C. Peng, C. Zhang, C. Peng, and J. Man, "A reinforcement learning approach to map reduce auto-configuration under networked environment," International Journal of Security and Networks, vol. 12, no. 3, pp. 135--140, 2017.

Digital Library

[46]

L. Bao, X. Liu, Z. Xu, and B. Fang, "Autoconfig: Automatic configuration tuning for distributed message systems." in Automated Software Engineering (ASE), 2018 33rd IEEE/ACM International Conference on. IEEE, 2018, pp. 29--40.

[47]

H. Chen, W. Zhang, and G. Jiang, "Experience transfer for the configuration tuning in large-scale computing systems," IEEE Transactions on Knowledge and Data Engineering, vol. 23, no. 3, pp. 388--401, 2011.

Digital Library

[48]

P. Jamshidi, N. Siegmund, M. Velez, and C. Kästner, "Transfer learning for performance modeling of configurable systems: An exploratory analysis," in Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 2017, pp. 497--508.

[49]

C. Blum and A. Roli, "Metaheuristics in combinatorial optimization: Overview and conceptual comparison," ACM computing surveys (CSUR), vol. 35, no. 3, pp. 268--308, 2003.

Digital Library

[50]

C. H. Papadimitriou and K. Steiglitz, "Combinatorial optimization: algorithms and complexity," 1982.

[51]

M. R. Garey and D. S. Johnson, "Computers and intractability: a guide to np-completeness," 1979.

[52]

P. I. Frazier, "A tutorial on bayesian optimization," arXiv preprint arXiv:1807.02811, 2018.

[53]

X. Li and X. Yao, "Cooperatively coevolving particle swarms for large scale optimization," IEEE Transactions on Evolutionary Computation, vol. 16, no. 2, pp. 210--224, 2012.

Digital Library

[54]

D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014.

[55]

S. Huang, J. Huang, J. Dai, T. Xie, and B. Huang, "The hibench benchmark suite: Characterization of the mapreduce-based data analysis," in Data Engineering Workshops (ICDEW), 2010 IEEE 26th International Conference on. IEEE, 2010, pp. 41--51.

[56]

B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears, "Benchmarking cloud serving systems with ycsb," in Proceedings of the 1st ACM symposium on Cloud computing. ACM, 2010, pp. 143--154.

[57]

JMeter, "https://jmeter.apache.org/," Accessed on July 31th 2018.

Cited By

Wang SLin BSun ZWen MLiu YLei YMao X(2023)Two Birds with One Stone: Boosting Code Generation and Code Search via a Generative Adversarial NetworkProceedings of the ACM on Programming Languages10.1145/36228157:OOPSLA2(486-515)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3622815
Chen TLi M(2022)Do Performance Aspirations Matter for Guiding Software Configuration Tuning? An Empirical Investigation under Dual Performance ObjectivesACM Transactions on Software Engineering and Methodology10.1145/357185332:3(1-41)Online publication date: 24-Nov-2022
https://dl.acm.org/doi/10.1145/3571853
Dou HWang YZhang YChen P(2022)DeepCAT: A Cost-Efficient Online Configuration Auto-Tuning Approach for Big Data FrameworksProceedings of the 51st International Conference on Parallel Processing10.1145/3545008.3545018(1-11)Online publication date: 29-Aug-2022
https://dl.acm.org/doi/10.1145/3545008.3545018
Show More Cited By

Recommendations

AutoConfig: automatic configuration tuning for distributed message systems
ASE '18: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering

Distributed message systems (DMSs) serve as the communication backbone for many real-time streaming data processing applications. To support the vast diversity of such applications, DMSs provide a large number of parameters to configure. However, It ...
BestConfig: tapping the performance potential of systems via automatic configuration tuning
SoCC '17: Proceedings of the 2017 Symposium on Cloud Computing

An ever increasing number of configuration parameters are provided to system users. But many users have used one configuration setting across different workloads, leaving untapped the performance potential of systems. A good configuration setting can ...
Ibm websphere portal for multiplatforms v5.1 handbook

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '19: Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering

November 2019

1333 pages

ISBN:9781728125084

General Chair:
Thomas Zimmermann
Microsoft Research
,
Program Chairs:
Julia Lawall
Inria/LIP6, France
,
Darko Marinov
University of Illinois at Urbana-Champaign

Sponsors

In-Cooperation

IEEE CS

Publisher

IEEE Press

Publication History

Published: 07 February 2020

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ASE '19

Sponsor:

ASE '19: 34nd IEEE/ACM International Conference on Automated Software Engineering

November 10 - 15, 2019

California, San Diego

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
71
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)2

Reflects downloads up to 04 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang SLin BSun ZWen MLiu YLei YMao X(2023)Two Birds with One Stone: Boosting Code Generation and Code Search via a Generative Adversarial NetworkProceedings of the ACM on Programming Languages10.1145/36228157:OOPSLA2(486-515)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3622815
Chen TLi M(2022)Do Performance Aspirations Matter for Guiding Software Configuration Tuning? An Empirical Investigation under Dual Performance ObjectivesACM Transactions on Software Engineering and Methodology10.1145/357185332:3(1-41)Online publication date: 24-Nov-2022
https://dl.acm.org/doi/10.1145/3571853
Dou HWang YZhang YChen P(2022)DeepCAT: A Cost-Efficient Online Configuration Auto-Tuning Approach for Big Data FrameworksProceedings of the 51st International Conference on Parallel Processing10.1145/3545008.3545018(1-11)Online publication date: 29-Aug-2022
https://dl.acm.org/doi/10.1145/3545008.3545018
Gong JChen TLo DMcIntosh SNovielli N(2022)Does configuration encoding matter in learning software performance?Proceedings of the 19th International Conference on Mining Software Repositories10.1145/3524842.3528431(482-494)Online publication date: 23-May-2022
https://dl.acm.org/doi/10.1145/3524842.3528431
He HJia ZLi SYu YZhou CLiao QWang JLiao XDwyer MDamian DZeller A(2022)Multi-intention-aware configuration selection for performance tuningProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510094(1431-1442)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510094

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents