research-article

Pythia: AI-assisted Code Completion System

Authors:

Alexey Svyatkovskiy,

Neel SundaresanAuthors Info & Claims

KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Pages 2727 - 2735

https://doi.org/10.1145/3292500.3330699

Published: 25 July 2019 Publication History

Abstract

In this paper, we propose a novel end-to-end approach for AI-assisted code completion called Pythia. It generates ranked lists of method and API recommendations which can be used by software developers at edit time. The system is currently deployed as part of Intellicode extension in Visual Studio Code IDE. Pythia exploits state-of-the-art large-scale deep learning models trained on code contexts extracted from abstract syntax trees. It is designed to work at a high throughput predicting the best matching code completions on the order of 100 ms.

We describe the architecture of the system, perform comparisons to frequency-based approach and invocation-based Markov Chain language model, and discuss challenges serving Pythia models on lightweight client devices.

The offline evaluation results obtained on 2700 Python open source software GitHub repositories show a top-5 accuracy of 92%, surpassing the baseline models by 20% averaged over classes, for both intra and cross-project settings.

Supplementary Material

MP4 File (p2727-svyatkovskiy.mp4)

Download
951.13 MB

References

[1]

A. Alnusair, T. Zhao, and E. Bodden. 2010. Effective API navigation and reuse. IRI (2010), 7--12.

[2]

Marcel Bruch, Martin Monperrus, and Mira Mezini. 2009. Learning from examples to improve code completion systems. In Proceedings of the the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering. ACM, 213--222.

Digital Library

[3]

R. I. Brafman G. Shani and D. Heckerman. 2002. An mdp-based recommender system. In Proceedings of 8th Conference on Uncertainty in Artificial Intelligence. UAI.

Digital Library

[4]

Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 1999. Learning to forget: Continual prediction with LSTM. (1999).

[5]

Priya Goyal, Piotr Dollá r, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. CoRR, Vol. abs/1706.02677 (2017). arxiv: 1706.02677 http://arxiv.org/abs/1706.02677

[6]

Alex Graves. 2013. Generating Sequences With Recurrent Neural Networks. CoRR, Vol. abs/1308.0850 (2013). arxiv: 1308.0850 http://arxiv.org/abs/1308.0850

[7]

T. Gvero, V. Kuncak, I. Kuraj, and R. Piskac. 2013. Complete completion using types and weights. In PLDI'13. ACM.

Digital Library

[8]

Hakan Inan, Khashayar Khosravi, and Richard Socher. 2016. Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling. CoRR, Vol. abs/1611.01462 (2016). arxiv: 1611.01462 http://arxiv.org/abs/1611.01462

[9]

Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. 2015. Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN). ICLR (2015).

[10]

M. Maraoiu, L. Church, and A. Blackwell. 2015. An empirical investigation of code completion usage by professional software developers. (2015).

[11]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 3111--3119. http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf

Digital Library

[12]

B. Mobasher, H. Dai, T. Luo, and M. Nakagawa. 2002. Using sequential and non-sequential patterns in predictive web usage mining tasks. In International Conference on Data Mining. IEEE.

Digital Library

[13]

G. C. Murphy, M. Kersten, and L. Findlater. 2006. How are Java software developers using the eclipse IDE? IEEE, Vol. 23, 4 (2006), 76--83.

Digital Library

[14]

Sebastian Proksch, Johannes Lerch, and Mira Mezini. 2015. Intelligent code completion with Bayesian networks. ACM Transactions on Software Engineering and Methodology (TOSEM), Vol. 25, 1 (2015), 3.

Digital Library

[15]

Dragomir R. Radev, Hong Qi, Harris Wu, and Weiguo Fan. 2002. Evaluating Web-based Question Answering Systems. Language Resources and Evaluation (2002).

[16]

Colin Raffel and Daniel P. W. Ellis. 2015. Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems. CoRR, Vol. abs/1512.08756 (2015). arxiv: 1512.08756 http://arxiv.org/abs/1512.08756

[17]

Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In ACM SIGPLAN Notices, Vol. 49. ACM, 419--428.

Digital Library

[18]

D'Souza Andrea Renika, Di Yang, and Cristina V Lopes. 2016. Collective Intelligence for Smarter API Recommendations in Python. In Source Code Analysis and Manipulation (SCAM), 2016 IEEE 16th International Working Conference on. IEEE, 51--60.

[19]

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Stroudsburg, PA, 1631--1642.

[20]

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 3104--3112. http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf

Digital Library

[21]

A. Zimdars, D. M. Chickering, and C. Meek. 2001. Using temporal data for making recommendations. In Proceedings of 7th Conference on Uncertainty in Artificial Intelligence. UAI, 580--588.

Digital Library

Cited By

Liu YYin YDeng JLi WPeng Z(2024)A Combinatorial Strategy for API Completion: Deep Learning and HeuristicsElectronics10.3390/electronics1318366913:18(3669)Online publication date: 15-Sep-2024
https://doi.org/10.3390/electronics13183669
Timofeev V(2024)Code Completion Using Markov ChainsVestnik NSU. Series: Information Technologies10.25205/1818-7900-2024-22-2-57-6722:2(57-67)Online publication date: 10-Oct-2024
https://doi.org/10.25205/1818-7900-2024-22-2-57-67
Weber TBrandmaier MSchmidt AMayer S(2024)Significant Productivity Gains through Programming with Large Language ModelsProceedings of the ACM on Human-Computer Interaction10.1145/36611458:EICS(1-29)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3661145
Show More Cited By

Index Terms

Pythia: AI-assisted Code Completion System
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Software and its engineering
  1. Software notations and tools
    1. Development frameworks and environments
      1. Integrated and visual development environments

Recommendations

IntelliCode compose: code generation using transformer
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

In software development through integrated development environments (IDEs), code completion is one of the most widely used features. Nevertheless, majority of integrated development environments only support completion of methods and APIs, or arguments. ...
Exploring and Improving Code Completion for Test Code
ICPC '24: Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension

Code completion is an important feature in Integrated Development Environments (IDEs). These years, researchers have been making efforts for intelligent code completion. However, existing work on intelligent code completion either only considered ...
Non-Autoregressive Line-Level Code Completion
Software developers frequently use code completion tools to accelerate software development by suggesting the following code elements. Researchers usually employ AutoRegressive (AR) decoders to complete code sequences in a left-to-right, token-by-token ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

July 2019

3305 pages

ISBN:9781450362016

DOI:10.1145/3292500

General Chairs:
Ankur Teredesai
KenSci
,
Vipin Kumar
University of Minnesota
,
Program Chairs:
Ying Li
EV Analysis Corporation
,
Rómer Rosales
LinkedIn
,
Evimaria Terzi
Boston University
,
George Karypis
University of Minnesota

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Microsoft

Conference

KDD '19

Sponsor:

KDD '19: The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 4 - 8, 2019

AK, Anchorage, USA

Acceptance Rates

KDD '19 Paper Acceptance Rate 110 of 1,200 submissions, 9%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

78
Total Citations
View Citations
1,776
Total Downloads

Downloads (Last 12 months)159
Downloads (Last 6 weeks)14

Reflects downloads up to 24 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu YYin YDeng JLi WPeng Z(2024)A Combinatorial Strategy for API Completion: Deep Learning and HeuristicsElectronics10.3390/electronics1318366913:18(3669)Online publication date: 15-Sep-2024
https://doi.org/10.3390/electronics13183669
Timofeev V(2024)Code Completion Using Markov ChainsVestnik NSU. Series: Information Technologies10.25205/1818-7900-2024-22-2-57-6722:2(57-67)Online publication date: 10-Oct-2024
https://doi.org/10.25205/1818-7900-2024-22-2-57-67
Weber TBrandmaier MSchmidt AMayer S(2024)Significant Productivity Gains through Programming with Large Language ModelsProceedings of the ACM on Human-Computer Interaction10.1145/36611458:EICS(1-29)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3661145
Chen SLi ZYang WLiu C(2024)DeciX: Explain Deep Learning Based Code Generation ApplicationsProceedings of the ACM on Software Engineering10.1145/36608141:FSE(2424-2446)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660814
Wang LWang QWang JZhao YWei MQuan ZCui DLi QChristakis MPradel M(2024)HECS: A Hypergraph Learning-Based System for Detecting Extract Class Refactoring OpportunitiesProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3685307(1851-1855)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3685307
Xia YLiang TMin WKuang LBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)Improving AST-Level Code Completion with Graph Retrieval and Multi-Field AttentionProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644420(125-136)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644420
Velasco APalacio DRodriguez-Cardenas DPoshyvanyk DRoychoudhury APaiva AAbreu RStorey MHierons RMadeira H(2024)Which Syntactic Capabilities Are Statistically Learned by Masked Language Models for Code?Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results10.1145/3639476.3639768(72-76)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639476.3639768
Li ZLi CTang ZHuang WGe JLuo BNg VWang THu YZhang X(2024)PTM-APIRec: Leveraging Pre-trained Models of Source Code in API RecommendationACM Transactions on Software Engineering and Methodology10.1145/363274533:3(1-30)Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1145/3632745
Xiao YSong WAhmed SGe XViswanath BMeng NYao D(2024)Measurement of Embedding Choices on Cryptographic API Completion TasksACM Transactions on Software Engineering and Methodology10.1145/362529133:3(1-30)Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1145/3625291
Aljohani ADo HHong JPark J(2024)From Fine-tuning to Output: An Empirical Investigation of Test Smells in Transformer-Based Test Code GenerationProceedings of the 39th ACM/SIGAPP Symposium on Applied Computing10.1145/3605098.3636058(1282-1291)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1145/3605098.3636058
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents