research-article

SmartSynth: synthesizing smartphone automation scripts from natural language

Authors:

Zhendong SuAuthors Info & Claims

MobiSys '13: Proceeding of the 11th annual international conference on Mobile systems, applications, and services

Pages 193 - 206

https://doi.org/10.1145/2462456.2464443

Published: 25 June 2013 Publication History

Abstract

This paper presents SmartSynth, a novel end-to-end programming system for synthesizing smartphone automation scripts from natural language descriptions. Our approach is unique in two key aspects. First, it involves a carefully designed domain-specific language that incorporates standard constructs from smartphone programming platforms to balance its expressivity and the ability to synthesize scripts from natural language. Second, our synthesis algorithm integrates techniques from two research areas: (1) It infers the set of components and their partial dataflow relations from the natural language description using techniques from the Natural Language Processing community; and (2) It uses techniques from the Program Synthesis community to infer missing dataflow relations via type-based synthesis and constructs scripts in a process akin to reverse parsing. SmartSynth also performs conversational interactions with the user when multiple top-ranked scripts exist or it cannot map part of the description to any component. Evaluated on 50 tasks collected from smartphone help forums, our system produces the intended scripts in real time for over 90% of the 640 natural language descriptions obtained from a user study for those tasks. SmartSynth has also been adapted to TouchDevelop, an end user-targeted programming environment on mobile platforms, with very promising results (see http://www.cs.ucdavis.edu/~su/smartsynth.mp4 for a video demo). We believe that SmartSynth is a step toward fully personalized use of smartphones' increasingly rich functionalities.

References

[1]

http://nlp.naturalparsing.com/.

[2]

A. V. Aho, M. S. Lam, R. Sethi, and J. D. Ullman. Compilers: Principles, Techniques, and Tools (2nd Edition). Addison Wesley, August 2006.

Digital Library

[3]

R. Alur, L. D'Antoni, S. Gulwani, D. Kini, and M. Viswanathan. Automated grading of dfa constructions. In IJCAI, 2013.

Digital Library

[4]

L. Androutsopoulos. Natural language interfaces to databases - an introduction. Journal of Natural Language Engineering, 1, 1995.

[5]

Apple. Siri for iPhone. http://www.apple.com/iphone/features/siri.html.

[6]

J. Aron. How innovative is Apple's new voice assistant, Siri? The New Scientist, 2011.

[7]

Crafty Apps. Tasker for Android. http://tasker.dinglisch.net/.

[8]

S. S. Epstein. Transportable natural language processing through simplicity--the PRE system. ACM Transactions on Information Systems (TOIS), 3(2), 1985.

Digital Library

[9]

C. Finucane, G. Jing, and H. Kress-Gazit. LTLMoP: Experimenting with language, temporal logic and robot control. In IEEE/RSJ ICIRS, 2010.

[10]

Google. Voice Actions for Android. http://www.google.com/mobile/voice-actions/.

[11]

S. Gulwani. Dimensions in program synthesis. In PPDP, 2010.

Digital Library

[12]

S. Gulwani. Automating string processing in spreadsheets using input-output examples. In POPL, 2011.

Digital Library

[13]

S. Gulwani. Synthesis from examples: Interaction models and algorithms. SYNASC, 2012. Invited talk paper.

Digital Library

[14]

S. Gulwani, W. Harris, and R. Singh. Spreadsheet data manipulation using examples. Communication of the ACM, 2012.

Digital Library

[15]

S. Gulwani, S. Jha, A. Tiwari, and R. Venkatesan. Synthesis of loop-free programs. In PLDI, 2011.

Digital Library

[16]

S. Gulwani, V. Korthikanti, and A. Tiwari. Synthesizing geometry constructions. In PLDI, 2011.

Digital Library

[17]

T. Gvero, V. Kuncak, and R. Piskac. Interactive Synthesis of Code Snippets. In CAV, 2011.

Digital Library

[18]

G. Hendrix, E. Sacerdoti, D. Sagalowicz, and J. Slocum. Developing a natural language interface to complex data. ACM Transactions on Database Systems (TODS), 3(2), 1978.

Digital Library

[19]

D. Jurafsky and J. H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics and Speech Recognition. Prentice Hall, 2nd edition, 2008.

Digital Library

[20]

R. J. Kate, Y. W. Wong, and R. J. Mooney. Learning to transform natural to formal languages. In AAAI, 2005.

Digital Library

[21]

D. Klein and C. D. Manning. Accurate unlexicalized parsing. In ACL, 2003.

Digital Library

[22]

Y. Li, H. Yang, and H. V. Jagadish. NaLIX: an interactive natural language interface for querying XML. In ICMD, 2005.

Digital Library

[23]

G. Little and R. C. Miller. Keyword programming in Java. In ASE, 2007.

Digital Library

[24]

G. Little and R. C. Miller. Keyword programming in Java. ASE, 2009.

Digital Library

[25]

H. Liu and H. Lieberman. Metafor: visualizing stories as code. In IEEE IUI, 2005.

Digital Library

[26]

H. Liu and H. Lieberman. Programmatic semantics for natural language interfaces. In Extended Abstracts on Human Factors in Computing Systems, 2005.

Digital Library

[27]

D. Mandelin, L. Xu, R. Bodík, and D. Kimelman. Jungloid mining: helping to navigate the api jungle. In PLDI, 2005.

Digital Library

[28]

M. D. Marneffe, B. MacCartney, and C. D. Manning. Generating typed dependency parses from phrase structure parses. In LREC, 2006.

[29]

Microsoft Research. on{X}. http://onx.ms/.

[30]

MIT Center for Mobile Learning. MIT App Inventor for Android. http://appinventor.mit.edu/.

[31]

R. Pandita, X. Xiao, H. Zhong, T. Xie, S. Oney, and A. Paradkar. Inferring method specifications from natural language api descriptions. In ICSE, 2012.

Digital Library

[32]

D. Perelman, S. Gulwani, T. Ball, and D. Grossman. Type-directed completion of partial expressions. In PLDI, 2012.

Digital Library

[33]

D. Price, E. Riloff, J. Zachary, and B. Harvey. NaturalJava: a natural language interface for programming in Java. In IEEE IUI, 2000.

Digital Library

[34]

M.-R. Ra, B. Liu, T. L. Porta, and R. Govindan. Medusa: A Programming Framework for Crowd-Sensing Applications. In MobiSys, 2012.

Digital Library

[35]

R. Singh and S. Gulwani. Learning semantic string transformations from examples. VLDB, 2012.

Digital Library

[36]

I. Song, F. Guedea, F. Karray, Y. Dai, and I. El Khalil. Natural language interface for mobile robot navigation control. In ISIC, 2004.

[37]

SRI International. Cognitive Assistant that Learns and Organizes. http://www.ai.sri.com/project/CALO.

[38]

N. Tillmann, M. Moskal, J. de Halleux, and M. Fahndrich. TouchDevelop: programming cloud-connected mobile devices via touchscreen. In ONWARD, 2011.

Digital Library

[39]

Two forty four a.m. LLC. Locale. http://www.twofortyfouram.com/.

[40]

N. Wirth. Program development by stepwise refinement. Communication of the ACM, 14(4), 1971.

Digital Library

[41]

X. Xiao, A. Paradkar, S. Thummalapenta, and T. Xie. Automated extraction of security policies from natural-language software documents. In FSE, 2012.

Digital Library

[42]

B. Yan and G. Chen. AppJoy: Appjoy: Personalized mobile application discovery. In MobiSys, 2011.

Digital Library

Cited By

Hatipoğlu ABilgin T(2024)DOĞAL DİL METİNLERİNDEN PROGRAMLAMA DİLİ KODU OLUŞTURMA ÇALIŞMALARI: BİR DERLEME ÇALIŞMASIİstanbul Ticaret Üniversitesi Fen Bilimleri Dergisi10.55071/ticaretfbd.1354040Online publication date: 21-Mar-2024
https://doi.org/10.55071/ticaretfbd.1354040
Kazemitabaar MChow JMa CEricson BWeintrop DGrossman T(2023)Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory ProgrammingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580919(1-23)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580919
Pidó SPinoli PCrovari PIeva FGarzotto FCeri S(2023)Ask Your Data—Supporting Data Science Processes by Combining AutoML and Conversational InterfacesIEEE Access10.1109/ACCESS.2023.327250311(45972-45988)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3272503
Show More Cited By

Index Terms

SmartSynth: synthesizing smartphone automation scripts from natural language
1. Software and its engineering
  1. Software notations and tools
    1. System description languages
      1. Design languages

Recommendations

Breaking Down High-Level Robot Path-Finding Abstractions in Natural Language Programming
AIxIA 2020 – Advances in Artificial Intelligence
Abstract
Natural language programming (NLPr) allows people to program in natural language (NL) for specific domains. It poses great potential since it gives non-experts the ability to develop projects without exhaustive training. However, complex ...
Interactive synthesis of temporal specifications from examples and natural language

Motivated by applications in robotics, we consider the task of synthesizing linear temporal logic (LTL) specifications based on examples and natural language descriptions. While LTL is a flexible, expressive, and unambiguous language to describe robotic ...
Can reactive synthesis and syntax-guided synthesis be friends?
SPLASH Companion 2021: Companion Proceedings of the 2021 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity

While reactive synthesis and syntax-guided synthesis (SyGuS) have seen enormous progress in recent years, combining the two approaches has remained a challenge. In this work, we present the synthesis of reactive programs from Temporal Stream Logic ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MobiSys '13: Proceeding of the 11th annual international conference on Mobile systems, applications, and services

June 2013

568 pages

ISBN:9781450316729

DOI:10.1145/2462456

General Chairs:
Hao-Hua Chu
National Taiwan University
,
Polly Huang
National Taiwan University
,
Program Chairs:
Romit Roy Choudhury
Duke University
,
Feng Zhao
Microsoft Research

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

In-Cooperation

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MobiSys'13

Sponsor:

SIGMOBILE

MobiSys'13: The 11th Annual International Conference on Mobile Systems, Applications, and Services

June 25 - 28, 2013

Taipei, Taiwan

Acceptance Rates

MobiSys '13 Paper Acceptance Rate 33 of 211 submissions, 16%;

Overall Acceptance Rate 274 of 1,679 submissions, 16%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

57
Total Citations
View Citations
471
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)3

Reflects downloads up to 05 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hatipoğlu ABilgin T(2024)DOĞAL DİL METİNLERİNDEN PROGRAMLAMA DİLİ KODU OLUŞTURMA ÇALIŞMALARI: BİR DERLEME ÇALIŞMASIİstanbul Ticaret Üniversitesi Fen Bilimleri Dergisi10.55071/ticaretfbd.1354040Online publication date: 21-Mar-2024
https://doi.org/10.55071/ticaretfbd.1354040
Kazemitabaar MChow JMa CEricson BWeintrop DGrossman T(2023)Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory ProgrammingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580919(1-23)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580919
Pidó SPinoli PCrovari PIeva FGarzotto FCeri S(2023)Ask Your Data—Supporting Data Science Processes by Combining AutoML and Conversational InterfacesIEEE Access10.1109/ACCESS.2023.327250311(45972-45988)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3272503
Costa Seco JAldrich JCarvalho LToninho BFerreira CScholliers CSinger J(2022)Derivations with Holes for Concept-Based Program SynthesisProceedings of the 2022 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software10.1145/3563835.3567658(63-79)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3563835.3567658
Young MNan ZShen XDwyer M(2022)IDE augmented with human-learning inspired natural language programmingProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings10.1145/3510454.3516832(110-114)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510454.3516832
Young MNan ZShen X(2022)IDE Augmented with Human-Learning Inspired Natural Language Programming2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)10.1109/ICSE-Companion55297.2022.9793806(110-114)Online publication date: May-2022
https://doi.org/10.1109/ICSE-Companion55297.2022.9793806
Nan ZShen XGuan H(2022)Enabling Near Real-Time NLU-Driven Natural Language Programming through Dynamic Grammar Graph-Based Translation2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO)10.1109/CGO53902.2022.9741262(278-289)Online publication date: 2-Apr-2022
https://doi.org/10.1109/CGO53902.2022.9741262
Sulír MPorubän J(2021)Natural mapping between voice commands and APIsOpen Computer Science10.1515/comp-2020-012511:1(135-145)Online publication date: 13-Jan-2021
https://doi.org/10.1515/comp-2020-0125
Rahmani KRaza MGulwani SLe VMorris DRadhakrishna ASoares GTiwari A(2021)Multi-modal program inference: a marriage of pre-trained language models and component-based synthesisProceedings of the ACM on Programming Languages10.1145/34855355:OOPSLA(1-29)Online publication date: 15-Oct-2021
https://dl.acm.org/doi/10.1145/3485535
Weigelt SSteurer VHey TTichy W(2020)Towards Programming in Natural Language: Learning New Functions from Spoken UtterancesInternational Journal of Semantic Computing10.1142/S1793351X2040009714:02(249-272)Online publication date: 23-Sep-2020
https://doi.org/10.1142/S1793351X20400097
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents