You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+10-13
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,7 @@
1
-
# CodeLLM Paper <imgsrc='https://img.shields.io/github/stars/PurCL/CodeLLMPaper'width="120"height="26" />
1
+
# CodeLLM Paper
2
2
3
3
This repository provides a curated list of research papers focused on Large Language Models (LLMs) for code. It aims to facilitate researchers and practitioners in exploring the rapidly growing body of literature on this topic. The papers are systematically collected from various top-tier venues, categorized, and labeled for easier navigation.
4
4
5
-
## 📍 News
6
-
-:star: We updated the papers published in CCS 2024, USENIX Sec 2024, and NDSS 2025. (03/05/2025)
7
-
8
5
## Table of Contents
9
6
10
7
-[A. Venues](#a-venues)
@@ -70,9 +67,9 @@ The papers in this repository are categorized along three dimensions: **Applicat
70
67
This category focuses on typical tasks in Software Engineering (SE) and Programming Languages (PL).
@@ -116,7 +113,7 @@ This category focuses on typical tasks in Software Engineering (SE) and Programm
116
113
117
114
This category concentrates on the LLMs' ability in understanding different forms of code and the non-functional properties of the LLMs (e.g., security and robustness). We also consider how to utilize the LLMs for general reasoning problems, such as typical agent-centric designs and specific PL designs for LLMs.
@@ -136,8 +133,8 @@ This category concentrates on the LLMs' ability in understanding different forms
136
133
137
134
This category includes studies on benchmarks, empirical evaluations, and surveys. The papers that do not belong to the following three categories are purely technical papers.
@@ -171,4 +168,4 @@ To facilitate timely batch updates to the paper repository, we prefer to utilize
171
168
172
169
This paper repository is intended solely for research purposes. All raw data is sourced from publicly available information on ACM, IEEE, and corresponding conference websites. Any content involving additional copyright information, including full PDF versions of the papers, is not disclosed in this repository.
"abstract": "In this paper, we tackle a critical challenge in model evaluation: how to keep code benchmarks useful when models might have already seen them during training. We introduce a novel solution, dynamic benchmarking framework, to address this challenge. Given a code understanding or reasoning benchmark, our framework dynamically transforms each input, i.e., programs, with various semantic-preserving mutations to build a syntactically new while semantically identical benchmark. We evaluated ten popular language models on our dynamic benchmarks. Our evaluation reveals several interesting or surprising findings: (1) all models perform significantly worse than before, (2) the ranking between some models shifts dramatically, and (3) our dynamic benchmarks can resist against the data contamination problem.",
6528
+
"labels": [
6529
+
"benchmark"
6530
+
],
6531
+
"url": "https://arxiv.org/abs/2503.06643"
6532
+
},
6533
+
"KNighter: Transforming Static Analysis with LLM-Synthesized Checkers": {
6534
+
"type": "article",
6535
+
"key": "knighter_2025",
6536
+
"title": "KNighter: Transforming Static Analysis with LLM-Synthesized Checkers",
"abstract": "Static analysis is a powerful technique for bug detection in critical systems like operating system kernels. However, designing and implementing static analyzers is challenging, timeconsuming, and typically limited to predefined bug patterns. While large language models (LLMs) have shown promise for static analysis, directly applying them to scan large codebases remains impractical due to computational constraints and contextual limitations. We present KNighter, the first approach that unlocks practical LLM-based static analysis by automatically synthesizing static analyzers from historical bug patterns. Rather than using LLMs to directly analyze massive codebases, our key insight is leveraging LLMs to generate specialized static analyzers guided by historical patch knowledge. KNighter implements this vision through a multi-stage synthesis pipeline that validates checker correctness against original patches and employs an automated refinement process to iteratively reduce false positives. Our evaluation on the Linux kernel demonstrates that KNighter generates high-precision checkers capable of detecting diverse bug patterns overlooked by existing human-written analyzers. To date, KNighter-synthesized checkers have discovered 70 new bugs/vulnerabilities in the Linux kernel, with 56 confirmed and 41 already fixed. 11 of these findings have been assigned CVE numbers. This work establishes an entirely new paradigm for scalable, reliable, and traceable LLM-based static analysis for real-world systems via checker synthesis.",
6542
+
"labels": [
6543
+
"static analysis",
6544
+
"bug detection"
6545
+
],
6546
+
"url": "https://arxiv.org/pdf/2503.09002"
6547
+
},
6548
+
"Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference": {
6549
+
"type": "article",
6550
+
"key": "knighter_2025",
6551
+
"title": "Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference",
6552
+
"author": "Thanh Le-Cong, Bach Le, Toby Murray",
6553
+
"journal": "arXiv preprint arXiv:2503.04779",
6554
+
"year": "2025",
6555
+
"venue": "arXiv2025",
6556
+
"abstract": "Large Language Models (LLMs) are increasingly being used to automate programming tasks. Yet, LLMs' capabilities in reasoning about program semantics are still inadequately studied, leaving significant potential for further exploration. This paper introduces FormalBench, a comprehensive benchmark designed to evaluate LLMs' reasoning abilities on program semantics, particularly via the task of synthesizing formal program specifications to assist verifying program correctness. This task requires both comprehensive reasoning over all possible program executions and the generation of precise, syntactically correct expressions that adhere to formal syntax and semantics. Using this benchmark, we evaluated the ability of LLMs in synthesizing consistent and complete specifications. Our findings show that LLMs perform well with simple control flows but struggle with more complex structures, especially loops, even with advanced prompting. Additionally, LLMs exhibit limited robustness against semantic-preserving transformations. We also highlight common failure patterns and design self-repair prompts, improving success rates by 25%.",
6557
+
"labels": [
6558
+
"static analysis",
6559
+
"specification inference",
6560
+
"benchmark",
6561
+
"empirical study"
6562
+
],
6563
+
"url": "https://arxiv.org/abs/2503.04779"
6564
+
},
6565
+
"Type-Aware Constraining for Code LLMs": {
6566
+
"type": "article",
6567
+
"key": "knighter_2025",
6568
+
"title": "Type-Aware Constraining for Code LLMs",
6569
+
"author": "Niels Mündler, Jingxuan He, Hao Wang, Koushik Sen, Dawn Song, Martin Vechev",
6570
+
"journal": "ICLR 2025 workshop",
6571
+
"year": "2025",
6572
+
"venue": "ICLR2025",
6573
+
"abstract": "Large Language Models (LLMs) have achieved notable success in code generation. However, they still frequently produce invalid code, as they do not precisely model formal aspects of programming languages. Constrained decoding is a promising approach to alleviate this issue and has been successfully applied to domain-specific languages and syntactic features, but is not able to enforce more semantic features, such as well-typedness. To address this issue, we introduce type-aware constrained decoding. We develop a novel prefix automata formalism and introduce a sound approach to guarantee existence of a type-safe completion of a partial program based on type inference and a search over inhabitable types. We implement type-aware constraining first for a foundational simply-typed language, then extend it to TypeScript. In our evaluation across state-of-the-art open-weight LLMs of up to 34B parameters and various model families, type-aware constraining reduces compilation errors by on average 70.9% and increases functional correctness by 16.2% in code synthesis, translation, and repair tasks.",
-[Hierarchical Repository-Level Code Summarization for Business Applications Using Local LLMs](../venues/arXiv2025/paper_7.md), ([arXiv2025](../venues/arXiv2025/README.md))
25
+
-[Hierarchical Repository-Level Code Summarization for Business Applications Using Local LLMs](../venues/arXiv2025/paper_10.md), ([arXiv2025](../venues/arXiv2025/README.md))
26
26
27
27
-**Abstract**: In large-scale software development, understanding the functionality and intent behind complex codebases is critical for effective development and maintenance. While code summarization has been widely studied, existing methods primarily focus on smaller code units, such as functions, and struggle with larger code artifacts like files and packages. Additionally, current summarization models tend to emphasize low-level implementation details, often overlooking the domain and business context that ...
Copy file name to clipboardExpand all lines: data/papers/labels/benchmark.md
+5
Original file line number
Diff line number
Diff line change
@@ -212,6 +212,11 @@
212
212
213
213
## Static Analysis
214
214
215
+
-[Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference](../venues/arXiv2025/paper_5.md), ([arXiv2025](../venues/arXiv2025/README.md))
216
+
217
+
-**Abstract**: Large Language Models (LLMs) are increasingly being used to automate programming tasks. Yet, LLMs' capabilities in reasoning about program semantics are still inadequately studied, leaving significant potential for further exploration. This paper introduces FormalBench, a comprehensive benchmark designed to evaluate LLMs' reasoning abilities on program semantics, particularly via the task of synthesizing formal program specifications to assist verifying program correctness. This task requires bo...
-[CompilerGym: robust, performant compiler optimization environments for AI research](../venues/CGO2022/paper_1.md), ([CGO2022](../venues/CGO2022/README.md))
216
221
217
222
-**Abstract**: Interest in applying Artificial Intelligence (AI) techniques to compiler optimizations is increasing rapidly, but compiler research has a high entry barrier. Unlike in other domains, compiler and AI researchers do not have access to the datasets and frameworks that enable fast iteration and development of ideas, and getting started requires a significant engineering investment. What is needed is an easy, reusable experimental infrastructure for real world compiler optimization tasks that can ser...
-[Combining Large Language Models with Static Analyzers for Code Review Generation](../venues/arXiv2025/paper_4.md), ([arXiv2025](../venues/arXiv2025/README.md))
63
+
-[Combining Large Language Models with Static Analyzers for Code Review Generation](../venues/arXiv2025/paper_7.md), ([arXiv2025](../venues/arXiv2025/README.md))
64
64
65
65
-**Abstract**: Code review is a crucial but often complex, subjective, and time-consuming activity in software development. Over the past decades, significant efforts have been made to automate this process. Early approaches focused on knowledge-based systems (KBS) that apply rule-based mechanisms to detect code issues, providing precise feedback but struggling with complex, context-dependent cases. More recent work has shifted toward fine-tuning pre-trained language models for code review, enabling broader is...
-[Evaluating the Effectiveness of Small Language Models in Detecting Refactoring Bugs](../venues/arXiv2025/paper_3.md), ([arXiv2025](../venues/arXiv2025/README.md))
123
+
-[Evaluating the Effectiveness of Small Language Models in Detecting Refactoring Bugs](../venues/arXiv2025/paper_6.md), ([arXiv2025](../venues/arXiv2025/README.md))
124
124
125
125
-**Abstract**: Popular IDEs frequently contain bugs in their refactoring implementations. Ensuring that a transformation preserves a program's behavior is a complex task. Traditional detection methods rely on predefined preconditions for each refactoring type, limiting their scalability and adaptability to new transformations. These methods often require extensive static and dynamic analyses, which are computationally expensive, time-consuming, and may still fail to detect certain refactoring bugs. This study ...
-[KNighter: Transforming Static Analysis with LLM-Synthesized Checkers](../venues/arXiv2025/paper_4.md), ([arXiv2025](../venues/arXiv2025/README.md))
202
+
203
+
-**Abstract**: Static analysis is a powerful technique for bug detection in critical systems like operating system kernels. However, designing and implementing static analyzers is challenging, timeconsuming, and typically limited to predefined bug patterns. While large language models (LLMs) have shown promise for static analysis, directly applying them to scan large codebases remains impractical due to computational constraints and contextual limitations. We present KNighter, the first approach that unlocks p...
-[LAMD: Context-driven Android Malware Detection and Classification with LLMs](../venues/arXiv2025/paper_1.md), ([arXiv2025](../venues/arXiv2025/README.md))
202
208
203
209
-**Abstract**: The rapid growth of mobile applications has escalated Android malware threats. Although there are numerous detection methods, they often struggle with evolving attacks, dataset biases, and limited explainability. Large Language Models (LLMs) offer a promising alternative with their zero-shot inference and reasoning capabilities. However, applying LLMs to Android malware detection presents two key challenges: (1)the extensive support code in Android applications, often spanning thousands of class...
0 commit comments