… code compared to expected code, including metrics like CodeBLEU [31] and ParaBLEU [44].
With advancements in … In a manner similar to pseudocode-to-code training using Machine …

… 2019. Spoc: search-based pseudocode to code. Advances in Neural Information Processing
Systems, 32. [7] Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and …

… in the popularity of large language models (LLMs) has also been accompanied by
significant advancements in code-generation LLMs (Code-LLMs). Prominent examples of such …

… We begin by describing pass@k (Kulal et al., 2019), the current main approach for … , Oded
Padon, Alex Aiken, and Percy S Liang. Spoc: Search-based pseudocode to code. In H. Wallach…

… While these advancements have enabled language models (LMs) to become highlyskilled …
2019). Lastly, our source code data is drawn from permissively licensed GitHub repositories …

… With the recent advances of code generation techniques based on Large Language
Models (LLMs), developers are … Advances in neural information processing systems 31 (2018). …

… Code generation aims to generate code in a general-purpose programming language, such
… -guided code generation,” a task that assesses the quality of both thought steps and code …

… a difficult task, even for the most advanced LLMs. Currently, there are no datasets to … 2019),
but they are geared towards code generation from some text input (docstrings or pseudocode…

… program veri cation are now within reach of codegenerating large language models. Likewise,
program veri cation systems can … performance using the pass@k metric [Kulal et al. 2019; …

In this paper, we propose a novel graph-based methodology to evaluate the functional
correctness of SQL generation. Conventional metrics for assessing SQL code generation, such as …