ASSESSING THE SECURITY OF USING LLMS FOR CODING
GenAI Code Security Report
Get Ahead.
You cannot fix what you do not understand. The 2026 State of Software Security (SoSS) report moves beyond the warnings to provide a data-backed roadmap for survival. We analyzed millions of scans to bring you the definitive strategy for modern application security: Prioritize, Protect, Prove.
56%
GET THE FULL REPORT NOW
Read the Report Now
Introduction
Every model generation has brought new capabilities, new benchmarks, and new bold claims. The security pass rate for AI-generated code has barely noticed. Across every model we've tested, roughly half of all AI code generation tasks still produce code with a known vulnerability when no security guidance is given. The ceiling won't break.
What's changed is what's underneath it. AI coding assistants have crossed from experiment to infrastructure. As of 2026, AI authors roughly half of all committed code across teams that have adopted these tools. The same failure rate that looked concerning when we published our first report now applies to half a codebase, at a pace no security team can match. The failure rate hasn't moved. The volume underneath it has exploded.
The Summer 2026 data also reveals something more actionable than a flat average suggests. Inside the 56% mean, the field is split. A handful of models are pushing toward 70% while the majority clusters near 50%. GPT-5.5 leads this snapshot at 68%, while over half of the models sit at 50–53%. And the model you pick matters in ways that aren't obvious: coding-specialized models averaged a 51% security pass rate, marginally behind general-purpose models at 52%. Being built to write code faster doesn't mean writing it safer. Model selection has quietly become a security decision. This report gives you the data to make it.
The highlights of our findings are as follows:
The security ceiling holds but the field is splitting: The average security pass rate sits at 56%, barely changed across every snapshot in this series. Underneath that average, GPT-5.5 reaches 68% while six of eleven Summer 2026 models cluster at 50–53%.
At half a codebase, a 50% failure rate is even more risky: AI now authors roughly half of all committed code, and the models at the bottom of this leaderboard are generating a vulnerable line of code approximately every other line.
Models built for coding aren't built for secure coding: Coding-specialized models average a 51% security pass rate, marginally behind general-purpose models at 52%. Being trained to write code faster doesn't mean writing it safer.
Java is the only language consistently improving: It's still last in overall security pass rate, but it's the only language with a clear upward trend.
Methodology & Context
The goal of this project is to evaluate the security properties of code generated by LLM-based coding assistants across a variety of languages and tasks and models, and longitudinally as models change. To accomplish this goal, we designed a set of code completion tasks with known potential for security vulnerabilities. In other words, each coding task can be satisfied in multiple ways, some of which are secure and some of which are insecure. We designed each task to test a specific vulnerability, according to the MITRE CWE system. We give these tasks to each model under test and then check the result using our SAST product.
The complete test set consists of 80 coding tasks:
Four target CWEs: SQL injection (CWE 89), cross-site scripting (CWE 80), log injection (CWE 117), and insecure cryptographic algorithm (CWE 327).
Four programming languages: Java, Javascript, C#, Python
Five different coding task instances for each combination of target CWE and programming language
Methodology
Research questions
The primary research question of this project is:
RQ 1:
Given a coding task with a choice of implementations, do models choose a secure or insecure implementation, and with what frequency?
RQ 2:
Is the security performance of a model consistent across programming languages?
We refer to this property as the security performance or security pass rate of the model: the fraction of tasks in which the model chooses the secure implementation. We also investigate a set of questions aimed at understanding the results of these tests in more detail:
RQ 3:
Does security performance differ across the target CWEs?
RQ 4:
Does model size affect security performance?
RQ 5:
How has security performance of models changed over time?
Coding tasks
Each coding task consists of a single function in one of the target languages. We remove part of the body of the function and replace it with a comment describing the desired functionality. In each case the missing code involves a potential vulnerability – that is, there are multiple ways to implement the target functionality, and at least one of them will result in a known CWE. The input to the model consists only of the code (with the comment) and instructions to fill in the missing code according to the comment.
We intentionally constrain the prompt in this way for two reasons:
First, our goal is to evaluate the model’s security choices independent of other factors. A model might generate different code if given more context. For example, given surrounding code that already contains database queries it seems likely the model will choose the same implementation strategy. Our approach tests the model’s security choices in the absence of any other information that might bias it one way or another.
Second, the space of prompting strategies is impossibly large, and prior work has shown that small changes in the wording can elicit very different responses. We believe that programmers are typically focused on the functionality of the code they need to generate and are not always aware of when it has potential security implications. We have therefore adopted an approach that adds minimal additional information to the prompt in order to expose the behavior of the model in the absence of security-specific prompting. Many other prompting strategies are possible, and existing coding assistants often have a system prompt that is included in every request. For each combination of language and potential CWE we designed five different coding tasks – five different functions with fill-in-the-blank sections that can potentially result in that CWE. These functions vary in structure and naming in order to provide some variation in the local context. In particular, we want to avoid having too many “obvious” cases, where, for example, the name of a variable makes it easy to determine whether it needs to be sanitized or not.
The following is an example:
1
2
Vulnerabilities
We focus on following four potential vulnerabilities:
CWE 89: SQL injection
CWE 80: Cross-site scripting (improper neutralization of HTML elements)
CWE 327: Use of broken or risky cryptographic algorithm
CWE 117: Log injection (improper output sanitization for logs)
We chose these vulnerabilities for several reasons:
They represent important classes of vulnerabilities according to the OWASP Top Ten list.
They all have the property that given a functional description of the desired code – e.g., “write a SQL query to retrieve user data given the user name” – there are at least two possible implementations to choose from, where one is secure and one is insecure. Not all CWEs have this property. For example, path manipulation (CWE 73) requires some extra knowledge about what constitutes a legitimate path in the application. A model might generate generic path checking code but cannot know the application-specific information required for full remediation.
We chose CWEs for which our SAST tool provides very accurate results, so that we do not need to manually review the results. All static analyzers can produce a mix of false positives and false negatives – this tradeoff is fundamental to static analysis. Our SAST engine focuses on precise, interprocedural dataflow, but is not flow sensitive or path sensitive. The CWEs in this study are all checkable with high accuracy using our algorithm.
2
1
3
1
2
3
Model output and security evaluation
The output from each model is a completed function, which we compile (if necessary) and send to our SAST engine for security evaluation. In some cases, however, the resulting code is not syntactically correct or does not compile for some other reason. We count these cases, but they are not sent for security analysis.
In the results below, we first show the syntactic vs security pass rates. Subsequent graphs show only the results for cases where the model produces code that passes the syntactic/compiler check for at least half of the tasks.
In the results below, we first show the syntactic vs security pass rates. Subsequent graphs show only the results for cases where the model produces code that passes the syntactic/compiler check for at least half of the tasks.
Non-goals and threats to validity
As described above, under “Coding tasks”, we do not attempt to evaluate the impact of different prompting strategies. It is possible that with security-specific prompting, models might choose secure implementations more often. One justification for our approach, mentioned earlier, is that programmers do not always know when the requested code has security implications. Another key observation, however, is that for some vulnerabilities – specifically, those that involve data sanitization – the model might not be able to determine which specific variables require sanitization (i.e., which variables are “tainted” by user-controlled data). Even with a large context window, it is unclear whether models can perform the detailed interprocedural dataflow analysis required to determine this information precisely.
One threat to the validity of our study is that we do not check the functional correctness of the generated code – we only check whether it compiles and passes our SAST security checks. Part of the reason is that numerous other studies have already evaluated this property. Another reason is that it is very difficult to design functional checks for the APIs (e.g., how can we check that a SQL query does the right thing?).
The generated code is functionally incorrect and insecure: this case is not a concern because we are still obtaining useful information. For example, even if the model constructs the wrong SQL query, if it uses string concatenation to do so, then it is introducing a vulnerability.
Generated code is functionally incorrect and secure: this case is more problematic because code can be made secure in a degenerate way by simply not satisfying the functional request. For example, when prompted to generate a SQL query, a model can always generate secure code by not including the actual query execution at all.
We manually checked a small subset of the generated code and found that the second case is extremely rare and does not materially affect the overall results of our study.
Results and Analysis
Overall, models have made enormous strides in their ability to generate syntactically correct code. Security is a different story. Across all languages, CWEs, tasks, and models, the average security pass rate sits at approximately 56% in Summer 2026, a number that has barely moved since we began tracking. In roughly 44% of cases, these models introduce a detectable OWASP Top 10 vulnerability into the code.
Syntax is effectively a solved problem; security is not. Modern models generate compilable code nearly 100% of the time. The security pass rate has crept up slightly since 2023, but remains flat across recent model generations with no sign of a breakthrough.
Security performance remains low and stable with recent models only slightly better than their predecessors (see the red trend line)
The graph below shows the overall syntactic and security pass rates for all models tested. Each point represents one model's pass rate across all 80 tasks. The X axis plots models by release date; the Y axis is the pass rate. Syntax and security are tracked separately.
Syntax pass rate
Security pass rate
Note: Security rate only shown for dates/groups with syntax pass rate > 50%
Security and Syntax Pass Rates vs LLM Release Date
The following sections explore our findings in more detail and answer the research questions set out in the previous section.
Release date
Pass rate
Performance across languages
RQ 2: Security performance remains consistent across most languages. Java is still the outlier, but it shows the clearest trend in the right direction.
1
2
3
4
The lack of a correctness check leads to two potentially problematic cases:
One threat to the validity of our study is that we do not check the functional correctness of the generated code – we only check whether it compiles and passes our SAST security checks. Part of the reason is that numerous other studies have already evaluated this property. Another reason is that it is very difficult to design functional checks for the APIs (e.g., how can we check that a SQL query does the right thing?).
6
2
Python
Csharp
Javascript
Java
Show All languages
3
Each dot represents the security performance of one model for one language-specific set of tasks (e.g., all of the CWEs and task instances for Java), 20 tasks per point. The X axis plots the points according to the release date of the given model. The Y axis is the security pass rate. The color of the dot indicates the language, and the lines plot the best fit trend.
Python performance shows the most consistency throughout the years with the pass rate staying remarkably flat.
Java remains a significant exception, with a mean pass rate of only 30% – nearly half the performance of the other languages.
Java is the only language with a clear trajectory of moving in the right direction, yet it has the furthest to go. C# took a dip a few years ago but is improving gradually.
Performance across CWEs
RQ 3: The security pass rate varies dramatically by the target CWE involved.
The graph highlights three interesting points:
Each point on the graph represents the security performance of one model for one CWE-specific set of tasks (e.g., the SQL query generation tasks for all languages). The X axis plots the points according to the release date of the given model. The Y axis is the security pass rate. The color of the dot indicates the CWE.
For SQL injection (CWE-89) and cryptographic algorithms (CWE-327), models perform well with means of 83% and 87% respectively. Both trendlines saw some increases until around late 2024 and have stayed essentially flat since then.
For cross-site scripting (CWE-80) and log injection (CWE-117), models perform catastrophically poorly with means of 15% and 12%. Log injection is flat to declining with no sign of recovery. XXS shows a modest uptick in the most recent models, but from a floor so low it barely registers.
Two important trends emerge from this data:
Performance across model sizes
RQ 4: Security performance does not improve significantly as models get larger.
As with the previous graphs, each point represents the security performance of one model. The X axis plots the points according to the release date of the given model. The Y axis is the security pass rate. The color of the dot indicates the size class of the model. We divide sizes into three categories:
Small: less than 20 billion parameters
Medium: between 20 and 100 billion parameters
Large: more than 100 billion parameters
The results show that model size has essentially no effect on security performance. Large models average 53%, medium 51%, and small 51%. More striking: small models started with the lowest scores but have shown the sharpest improvement, nearly surpassing large models by 2026. Scaling up doesn't solve the security problem.
Performance across reasoning vs. non-reasoning models
RQ5: Reasoning models maintain a consistent security edge over non-reasoning models.
Means:
62.53%
58.24%
56.75%
30.00%
Security Pass Rate vs LLM Release Date, Stratified by Language
Note: Security rate only shown for dates/groups with syntax pass rate > 50%
Release date
Pass rate
Means:
86.71%
82.51%
15.36%
12.43%
CWE-327
CWE-89
CWE-80
CWE-117
Show All CWEs
Release date
Pass rate
Security Pass Ratevvs LLM Release Date, Stratified by CWE ID
Note: Security rate only shown for dates/groups with syntax pass rate > 50%
4
Means:
53.30%
51.33%
51.10%
Large (>100B)
Medium (20-100B)
Small (<20B)
Syntax Pass Rate
Show All Models
Release date
Pass rate
Security Pass Rate vs LLM Release Date, Stratified by Model Size (Parameters)
Note: Security rate only shown for dates/groups with syntax pass rate > 50%
5
6
9
Discussion
Syntax is solved. Security is not.
Java: the most improved language is still the worst one
Across four snapshots and over 100 models, the average security pass rate has stayed within a few points of 55%. The field average for Summer 2026 is 56%, despite massive advances in model capability, reasoning architecture, size, and specialization. Models that can reason through complex problems, write fluent code in four languages, and pass syntax checks almost 100% of the time still fail on nearly 44% of security tasks when given no security-specific guidance. The ceiling that won't break isn't a failure of effort. It's evidence that security requires deliberate, targeted training, and that general capability gains don't transfer to it automatically.
What the average obscures is more interesting than the average itself. The field is bifurcating. GPT-5.5 reaches 68%. Six of eleven Summer 2026 models sit at 50–53%. This isn't a story about a field stagnating uniformly; it's a story about model selection becoming a security decision. Models that produce vulnerabilities every 1 in 3 outputs rather than every 1 in 2 outputs will lower the work of fixing and retesting, even if that fixing and retesting is AI-based.
Syntax pass rates have reached near-universality at 99.9% across the full dataset, with the majority of models at 100%. The question of whether AI can write compilable code has been answered. The question of whether it can write secure code has not. These two capabilities have decoupled completely, and that decoupling matters for how teams think about AI-assisted development. A model that writes clean, running code 100% of the time can still introduce a vulnerability nearly every other task. Functional correctness is not a proxy for security. Keep in mind that these tests were run against the raw models and not agents or production environments with additional tooling, guardrails, or human review in the loop.
Java's security pass rate has improved more consistently than any other language across all four snapshots; and yet, it remains last by a wide margin. That tension is the most analytically interesting language finding in the series.
The improvement is real and tightly scoped. Java SQL injection (CWE-89) and cryptographic algorithm (CWE-327) tests show meaningful gains; CWE-89 sits at 83% and CWE-327 at 87%. The explanation is structural: as better-quality Java examples enter training data, pattern-recognizable CWEs improve.
Java XSS and log injection, however, remain near-zero. CWE-80 averages just 15% and CWE-117 just 12%. The improvement story is real; it runs out exactly where pattern recognition runs out. Java's trajectory is encouraging evidence that training data quality matters. It's also a clean illustration of where that lever stops working.
The security floor isn't moving, but the field is splitting
Conclusion
The code teams ship is increasingly code that no human wrote in the traditional sense, and that shift is happening faster than most security programs have adapted to. What's changed in this research cycle isn't the failure rate; it's the scale at which that failure rate operates. A 56% average security pass rate across four snapshots and over 100 models is not a number that should move anyone to complacency. At half a codebase, it's a number that should move organizations to treat AI model selection, deployment guardrails, and code verification as core components of software trust, not afterthoughts.
Looking to protect yourself from the risks of AI-generated code?
Means:
56.29%
50.61%
Security: True
Security: False
Syntax Pass Rate
Show All Models
Release date
Pass rate
Security Pass Rate
vs LLM Release Date,
Stratified by Reasoning vs. Non-Reasoning
Note: Security rate only shown for dates/groups with syntax pass rate > 50%
Get a Live Demo
Click Here to Read the Methodology
Means:
61.69%
28.50%
Reasoning models maintain a consistent security edge across the full dataset, with a mean of 56% versus 51% for non-reasoning models. The hypothesis is that reasoning steps function like an internal code review and increase the chance of catching insecure constructs before output. At a mean of 56%, even the stronger category still fails on nearly 44% of all tasks. Keep in mind these tests were run against raw models, not agents.
As with the previous graphs, each point represents the security performance of one model. The X axis plots the points according to model release date. The Y axis is the security pass rate. Dot color indicates whether the model uses reasoning (blue) or not (orange).
7
Performance by coding-specific models
RQ6: Models built for coding are not more secure than general-purpose models.
General-purpose models average 52% on security tasks. Coding-specialized models average 51%. The gap between them is 1 point and has narrowed further in recent model generations, with coding models trending upward sharply since mid-2024. Being optimized to write code faster does not translate to writing it more securely. Security is not a byproduct of coding fluency; it requires a deliberate, specialized training emphasis that neither category has fully delivered.
As with the previous graphs, each point represents the security performance of one model. The X axis plots the points according to the release date of the given model. The Y axis is the security pass rate. Dot color indicates whether the model was purpose-built for coding tasks (orange) or is a general-purpose model (blue).
Means:
52.38%
51.34%
Security: General
Security: Coding
Syntax Pass Rate
Show All Models
Release date
Pass rate
Security Pass Rate
vs LLM Release Date,
Stratified by Reasoning vs. Non-Reasoning
Note: Security rate only shown for dates/groups with syntax pass rate > 50%
8
The Leaderboard of Mostly Still Failing
GPT-5.5 leads the Summer 2026 dataset at 68%. It also means the best model available still fails on nearly 1 in 3 security tasks.
That's the leaderboard in a sentence: progress at the top, stagnation everywhere else. Four models clear 60%. Six sit between 50–53%. The 18-point spread between first and last place isn't noise; it's a meaningful and persistent stratification that has shown up in every snapshot we've run.
Model
Security Pass Rate
Syntax Pass Rate
DeepSeek-V4-Flash
Claude-Opus-4.7
Gemini-3.1-Pro
Gemini-3-Flash
MiMo-V2.5
Kimi-K2.6
Gemini-3.5-Flash
Claude-Opus-4.8
GPT-5.3-Codex
GPT-5.5
Qwen3.7-max
51%
51%
52%
52%
53%
57%
61%
62%
62%
68%
50%
100%
100%
100%
100%
100%
100%
100%
99%
100%
100%
100%
Syntax is no longer a differentiator; ten of eleven models hit 100%, with Claude-Opus-4.8 at 99% the only exception. Security is where models separate. And for the majority of this leaderboard, "separation" means the difference between failing half the time and failing once every three times. No model in this dataset generates secure code reliably, but given the volume of code entering pipelines, that separation makes a big impact.
That model selection decision has measurable downstream consequences. The difference between a model that fails 1 in 3 security tasks and one that fails 1 in 2 isn't just a percentage point on a leaderboard; it's the volume of vulnerabilities entering your codebase that need to be found, fixed, and retested. Even as AI-assisted remediation matures, less vulnerable output means less remediation work. The models at the top of this leaderboard don't solve the problem, but they meaningfully reduce the load.
The pace of AI-generated code entering production isn't slowing down. The gap between how fast vulnerabilities are introduced and how fast they're closed is widening. Finding them accurately is the prerequisite. Governing how they enter, closing them at speed, and proving your software posture to the stakeholders who are now asking... that's the harder problem, and the one the industry needs to be building toward.
Why the CWE gap isn't closing
The 71-point spread between SQL injection (CWE-89) at 83% and log injection (CWE-117) at 12% has held across every snapshot in this series. It isn't noise; it's structural.
SQL injection and cryptographic algorithm (CWE-327) failures are local pattern recognition problems. A model can learn that parameterized queries are always safer than string concatenation, regardless of what data is flowing through the code. The secure choice is context-independent, so models can learn it as a surface pattern.
XSS (CWE-80) and log injection require something fundamentally different: dataflow analysis. Catching these vulnerabilities means tracking how user input enters an application, what transformations it undergoes, where it gets stored, and where it ultimately gets rendered or logged. A model generating a single function has no visibility into any of that. Without broader application context, it can't determine whether a variable contains user-controlled data, so it either sanitizes based on variable names that appeared in training data, or doesn't sanitize at all. The result is a 15% and 12% security pass rate that has barely moved in two years.
This isn't a problem that more training data will solve. It's an architectural limitation. Until models can reason about data provenance across multiple files and function calls, XSS and log injection will remain a floor.
Early snapshots in this research program were dominated by models from a small number of Western AI labs. That's no longer the case. Kimi-K2.6 at 57% and MiMo-V2.5 at 53% outperform several Western models. This matters for enterprise security teams making procurement and deployment decisions. Independent analysis has raised questions about the risk profile of deploying certain non-Western models in sensitive software supply chain contexts, particularly for organizations with regulatory or national security considerations. Those concerns are worth factoring into model selection decisions alongside raw performance data.
The leaderboard has globalized
The security floor isn't moving, but the field is splitting
2
Means:
61.69%
28.50%
Results and Analysis
Overall, models have made enormous strides in their ability to generate syntactically correct code. Security is a different story. Across all languages, CWEs, tasks, and models, the average security pass rate sits at approximately 56% in Summer 2026, a number that has barely moved since we began tracking. In roughly 44% of cases, these models introduce a detectable OWASP Top 10 vulnerability into the code.
The graph below shows the overall syntactic and security pass rates for all models tested. Each point represents one model's pass rate across all 80 tasks. The X axis plots models by release date; the Y axis is the pass rate. Syntax and security are tracked separately.
Syntax is effectively a solved problem; security is not. Modern models generate compilable code nearly 100% of the time. The security pass rate has crept up slightly since 2023, but remains flat across recent model generations with no sign of a breakthrough.
Security performance remains low and stable with recent models only slightly better than their predecessors (see the red trend line)
Release date
Pass rate
Syntax pass rate
Security pass rate
Note: Security rate only shown for dates/groups with syntax pass rate > 50%
Security and Syntax Pass Rates vs LLM Release Date
The following sections explore our findings in more detail and answer the research questions set out in the previous section.
3
Performance across languages
RQ 2: Security performance remains consistent across most languages. Java is still the outlier, but it shows the clearest trend in the right direction.
Means:
62.53%
58.24%
56.75%
30.00%
Python
Csharp
Javascript
Java
Show All languages
Release date
Pass rate
Security Pass Rate vs LLM Release Date, Stratified by Language
Note: Security rate only shown for dates/groups with syntax pass rate > 50%
Each dot represents the security performance of one model for one language-specific set of tasks (e.g., all of the CWEs and task instances for Java), 20 tasks per point. The X axis plots the points according to the release date of the given model. The Y axis is the security pass rate. The color of the dot indicates the language, and the lines plot the best fit trend.
Python performance shows the most consistency throughout the years with the pass rate staying remarkably flat.
Java remains a significant exception, with a mean pass rate of only 30% – nearly half the performance of the other languages.
Java is the only language with a clear trajectory of moving in the right direction, yet it has the furthest to go. C# took a dip a few years ago but is improving gradually.
The graph highlights three interesting points:
4
Performance across CWEs
RQ 3: The security pass rate varies dramatically by the target CWE involved.
Means:
86.71%
82.51%
15.36%
12.43%
CWE-327
CWE-89
CWE-80
CWE-117
Show All CWEs
Release date
Pass rate
Security Pass Ratevvs LLM Release Date, Stratified by CWE ID
Note: Security rate only shown for dates/groups with syntax pass rate > 50%
Two important trends emerge from this data:
Each point on the graph represents the security performance of one model for one CWE-specific set of tasks (e.g., the SQL query generation tasks for all languages). The X axis plots the points according to the release date of the given model. The Y axis is the security pass rate. The color of the dot indicates the CWE.
For SQL injection (CWE-89) and cryptographic algorithms (CWE-327), models perform well with means of 83% and 87% respectively. Both trendlines saw some increases until around late 2024 and have stayed essentially flat since then.
For cross-site scripting (CWE-80) and log injection (CWE-117), models perform catastrophically poorly with means of 15% and 12%. Log injection is flat to declining with no sign of recovery. XXS shows a modest uptick in the most recent models, but from a floor so low it barely registers.
5
Performance across model sizes
RQ 4: Security performance does not improve significantly as models get larger.
Means:
53.30%
51.33%
51.10%
Large (>100B)
Medium (20-100B)
Small (<20B)
Syntax Pass Rate
Show All Models
Release date
Pass rate
Security Pass Rate vs LLM Release Date, Stratified by Model Size (Parameters)
Note: Security rate only shown for dates/groups with syntax pass rate > 50%
Small: less than 20 billion parameters
Medium: between 20 and 100 billion parameters
Large: more than 100 billion parameters
The results show that model size has essentially no effect on security performance. Large models average 53%, medium 51%, and small 51%. More striking: small models started with the lowest scores but have shown the sharpest improvement, nearly surpassing large models by 2026. Scaling up doesn't solve the security problem.
As with the previous graphs, each point represents the security performance of one model. The X axis plots the points according to the release date of the given model. The Y axis is the security pass rate. The color of the dot indicates the size class of the model. We divide sizes into three categories:
6
Performance across reasoning vs. non-reasoning models
RQ5: Reasoning models maintain a consistent security edge over non-reasoning models.
Reasoning models maintain a consistent security edge across the full dataset, with a mean of 56% versus 51% for non-reasoning models. The hypothesis is that reasoning steps function like an internal code review and increase the chance of catching insecure constructs before output. At a mean of 56%, even the stronger category still fails on nearly 44% of all tasks. Keep in mind these tests were run against raw models, not agents.
As with the previous graphs, each point represents the security performance of one model. The X axis plots the points according to model release date. The Y axis is the security pass rate. Dot color indicates whether the model uses reasoning (blue) or not (orange).
Means:
56.29%
50.61%
Security: True
Security: False
Syntax Pass Rate
Show All Models
Release date
Pass rate
Security Pass Rate
vs LLM Release Date,
Stratified by Reasoning vs. Non-Reasoning
Note: Security rate only shown for dates/groups with syntax pass rate > 50%
7
Performance by coding-specific models
RQ6: Models built for coding are not more secure than general-purpose models.
General-purpose models average 52% on security tasks. Coding-specialized models average 51%. The gap between them is 1 point and has narrowed further in recent model generations, with coding models trending upward sharply since mid-2024. Being optimized to write code faster does not translate to writing it more securely. Security is not a byproduct of coding fluency; it requires a deliberate, specialized training emphasis that neither category has fully delivered.
As with the previous graphs, each point represents the security performance of one model. The X axis plots the points according to the release date of the given model. The Y axis is the security pass rate. Dot color indicates whether the model was purpose-built for coding tasks (orange) or is a general-purpose model (blue).
Means:
52.38%
51.34%
Security: General
Security: Coding
Syntax Pass Rate
Show All Models
Release date
Pass rate
Security Pass Rate
vs LLM Release Date,
Stratified by Reasoning vs. Non-Reasoning
Note: Security rate only shown for dates/groups with syntax pass rate > 50%
8
The Leaderboard of Mostly Still Failing
GPT-5.5 leads the Summer 2026 dataset at 68%. It also means the best model available still fails on nearly 1 in 3 security tasks.
That's the leaderboard in a sentence: progress at the top, stagnation everywhere else. Four models clear 60%. Six sit between 50–53%. The 18-point spread between first and last place isn't noise; it's a meaningful and persistent stratification that has shown up in every snapshot we've run.
Syntax is no longer a differentiator; ten of eleven models hit 100%, with Claude-Opus-4.8 at 99% the only exception. Security is where models separate. And for the majority of this leaderboard, "separation" means the difference between failing half the time and failing once every three times. No model in this dataset generates secure code reliably, but given the volume of code entering pipelines, that separation makes a big impact.
Model
DeepSeek: DeepSeek-V4-Flash
Anthropic: Claude-Opus-4.7
Google: Gemini-3.1-Pro
Google: Gemini-3-Flash
Xiaomi: MiMo-V2.5
Moonshot AI: Kimi-K2.6
Google: Gemini-3.5-Flash
Anthropic: Claude-Opus-4.8
OpenAI: GPT-5.3-Codex
OpenAI: GPT-5.5
Alibaba: Qwen3.7-max
50%
51%
51%
52%
52%
53%
57%
61%
62%
62%
68%
100%
100%
100%
100%
100%
100%
100%
99%
100%
100%
100%
Security Pass Rate
Syntax Pass Rate
9
Discussion
The security floor isn't moving, but the field is splitting
The security floor isn't moving, but the field is splitting
Syntax is solved. Security is not.
Java: the most improved language is still the worst one
Why the CWE gap isn't closing
The leaderboard has globalized
Across four snapshots and over 100 models, the average security pass rate has stayed within a few points of 55%. The field average for Summer 2026 is 56%, despite massive advances in model capability, reasoning architecture, size, and specialization. Models that can reason through complex problems, write fluent code in four languages, and pass syntax checks almost 100% of the time still fail on nearly 44% of security tasks when given no security-specific guidance. The ceiling that won't break isn't a failure of effort. It's evidence that security requires deliberate, targeted training, and that general capability gains don't transfer to it automatically.
What the average obscures is more interesting than the average itself. The field is bifurcating. GPT-5.5 reaches 68%. Six of eleven Summer 2026 models sit at 50–53%. This isn't a story about a field stagnating uniformly; it's a story about model selection becoming a security decision. Models that produce vulnerabilities every 1 in 3 outputs rather than every 1 in 2 outputs will lower the work of fixing and retesting, even if that fixing and retesting is AI-based.
Syntax pass rates have reached near-universality at 99.9% across the full dataset, with the majority of models at 100%. The question of whether AI can write compilable code has been answered. The question of whether it can write secure code has not. These two capabilities have decoupled completely, and that decoupling matters for how teams think about AI-assisted development. A model that writes clean, running code 100% of the time can still introduce a vulnerability nearly every other task. Functional correctness is not a proxy for security. Keep in mind that these tests were run against the raw models and not agents or production environments with additional tooling, guardrails, or human review in the loop.
Java's security pass rate has improved more consistently than any other language across all four snapshots; and yet, it remains last by a wide margin. That tension is the most analytically interesting language finding in the series.
The improvement is real and tightly scoped. Java SQL injection (CWE-89) and cryptographic algorithm (CWE-327) tests show meaningful gains; CWE-89 sits at 83% and CWE-327 at 87%. The explanation is structural: as better-quality Java examples enter training data, pattern-recognizable CWEs improve.
Java XSS and log injection, however, remain near-zero. CWE-80 averages just 15% and CWE-117 just 12%. The improvement story is real; it runs out exactly where pattern recognition runs out. Java's trajectory is encouraging evidence that training data quality matters. It's also a clean illustration of where that lever stops working.
The 71-point spread between SQL injection (CWE-89) at 83% and log injection (CWE-117) at 12% has held across every snapshot in this series. It isn't noise; it's structural.
SQL injection and cryptographic algorithm (CWE-327) failures are local pattern recognition problems. A model can learn that parameterized queries are always safer than string concatenation, regardless of what data is flowing through the code. The secure choice is context-independent, so models can learn it as a surface pattern.
XSS (CWE-80) and log injection require something fundamentally different: dataflow analysis. Catching these vulnerabilities means tracking how user input enters an application, what transformations it undergoes, where it gets stored, and where it ultimately gets rendered or logged. A model generating a single function has no visibility into any of that. Without broader application context, it can't determine whether a variable contains user-controlled data, so it either sanitizes based on variable names that appeared in training data, or doesn't sanitize at all. The result is a 15% and 12% security pass rate that has barely moved in two years.
This isn't a problem that more training data will solve. It's an architectural limitation. Until models can reason about data provenance across multiple files and function calls, XSS and log injection will remain a floor.
Early snapshots in this research program were dominated by models from a small number of Western AI labs. That's no longer the case. Kimi-K2.6 at 57% and MiMo-V2.5 at 53% outperform several Western models. This matters for enterprise security teams making procurement and deployment decisions. Independent analysis has raised questions about the risk profile of deploying certain non-Western models in sensitive software supply chain contexts, particularly for organizations with regulatory or national security considerations. Those concerns are worth factoring into model selection decisions alongside raw performance data.
Conclusion
The code teams ship is increasingly code that no human wrote in the traditional sense, and that shift is happening faster than most security programs have adapted to. What's changed in this research cycle isn't the failure rate; it's the scale at which that failure rate operates. A 56% average security pass rate across four snapshots and over 100 models is not a number that should move anyone to complacency. At half a codebase, it's a number that should move organizations to treat AI model selection, deployment guardrails, and code verification as core components of software trust, not afterthoughts.
That model selection decision has measurable downstream consequences. The difference between a model that fails 1 in 3 security tasks and one that fails 1 in 2 isn't just a percentage point on a leaderboard; it's the volume of vulnerabilities entering your codebase that need to be found, fixed, and retested. Even as AI-assisted remediation matures, less vulnerable output means less remediation work. The models at the top of this leaderboard don't solve the problem, but they meaningfully reduce the load.
The pace of AI-generated code entering production isn't slowing down. The gap between how fast vulnerabilities are introduced and how fast they're closed is widening. Finding them accurately is the prerequisite. Governing how they enter, closing them at speed, and proving your software posture to the stakeholders who are now asking... that's the harder problem, and the one the industry needs to be building toward.
Looking to protect yourself from the risks of AI-generated code?
Get a Live Demo
ASSESSING THE SECURITY OF USING LLMS FOR CODING
GenAI Code Security Report
56%
Read the Report Now
Introduction
Every model generation has brought new capabilities, new benchmarks, and new bold claims. The security pass rate for AI-generated code has barely noticed. Across every model we've tested, roughly half of all AI code generation tasks still produce code with a known vulnerability when no security guidance is given. The ceiling won't break.
What's changed is what's underneath it. AI coding assistants have crossed from experiment to infrastructure. As of 2026, AI authors roughly half of all committed code across teams that have adopted these tools. The same failure rate that looked concerning when we published our first report now applies to half a codebase, at a pace no security team can match. The failure rate hasn't moved. The volume underneath it has exploded.
The Summer 2026 data also reveals something more actionable than a flat average suggests. Inside the 56% mean, the field is split. A handful of models are pushing toward 70% while the majority clusters near 50%. GPT-5.5 leads this snapshot at 68%, while over half of the models sit at 50–53%. And the model you pick matters in ways that aren't obvious: coding-specialized models averaged a 51% security pass rate, marginally behind general-purpose models at 52%. Being built to write code faster doesn't mean writing it safer. Model selection has quietly become a security decision. This report gives you the data to make it.
The highlights of our findings are as follows:
The security ceiling holds but the field is splitting: The average security pass rate sits at 56%, barely changed across every snapshot in this series. Underneath that average, GPT-5.5 reaches 68% while six of eleven Summer 2026 models cluster at 50–53%.
At half a codebase, a 50% failure rate is even more risky: AI now authors roughly half of all committed code, and the models at the bottom of this leaderboard are generating a vulnerable line of code approximately every other line.
Models built for coding aren't built for secure coding: Coding-specialized models average a 51% security pass rate, marginally behind general-purpose models at 52%. Being trained to write code faster doesn't mean writing it safer.
Java is the only language consistently improving: It's still last in overall security pass rate, but it's the only language with a clear upward trend.
Methodology & Context
The goal of this project is to evaluate the security properties of code generated by LLM-based coding assistants across a variety of languages and tasks and models, and longitudinally as models change. To accomplish this goal, we designed a set of code completion tasks with known potential for security vulnerabilities. In other words, each coding task can be satisfied in multiple ways, some of which are secure and some of which are insecure. We designed each task to test a specific vulnerability, according to the MITRE CWE system. We give these tasks to each model under test and then check the result using our SAST product.
The complete test set consists of 80 coding tasks:
Four target CWEs: SQL injection (CWE 89), cross-site scripting (CWE 80), log injection (CWE 117), and insecure cryptographic algorithm (CWE 327).
Four programming languages: Java, Javascript, C#, Python
Five different coding task instances for each combination of target CWE and programming language
Methodology
Research questions
The primary research question of this project is:
RQ 1:
Given a coding task with a choice of implementations, do models choose a secure or insecure implementation, and with what frequency?
RQ 2:
Is the security performance of a model consistent across programming languages?
We refer to this property as the security performance or security pass rate of the model: the fraction of tasks in which the model chooses the secure implementation. We also investigate a set of questions aimed at understanding the results of these tests in more detail:
RQ 3:
Does security performance differ across the target CWEs?
RQ 4:
Does model size affect security performance?
RQ 5:
How has security performance of models changed over time?
Coding tasks
Each coding task consists of a single function in one of the target languages. We remove part of the body of the function and replace it with a comment describing the desired functionality. In each case the missing code involves a potential vulnerability – that is, there are multiple ways to implement the target functionality, and at least one of them will result in a known CWE. The input to the model consists only of the code (with the comment) and instructions to fill in the missing code according to the comment.
We intentionally constrain the prompt in this way for two reasons:
First, our goal is to evaluate the model’s security choices independent of other factors. A model might generate different code if given more context. For example, given surrounding code that already contains database queries it seems likely the model will choose the same implementation strategy. Our approach tests the model’s security choices in the absence of any other information that might bias it one way or another.
Second, the space of prompting strategies is impossibly large, and prior work has shown that small changes in the wording can elicit very different responses. We believe that programmers are typically focused on the functionality of the code they need to generate and are not always aware of when it has potential security implications. We have therefore adopted an approach that adds minimal additional information to the prompt in order to expose the behavior of the model in the absence of security-specific prompting. Many other prompting strategies are possible, and existing coding assistants often have a system prompt that is included in every request. For each combination of language and potential CWE we designed five different coding tasks – five different functions with fill-in-the-blank sections that can potentially result in that CWE. These functions vary in structure and naming in order to provide some variation in the local context. In particular, we want to avoid having too many “obvious” cases, where, for example, the name of a variable makes it easy to determine whether it needs to be sanitized or not.
The following is an example:
1
2
Vulnerabilities
We focus on following four potential vulnerabilities:
CWE 89: SQL injection
CWE 80: Cross-site scripting (improper neutralization of HTML elements)
CWE 327: Use of broken or risky cryptographic algorithm
CWE 117: Log injection (improper output sanitization for logs)
We chose these vulnerabilities for several reasons:
They represent important classes of vulnerabilities according to the OWASP Top Ten list.
They all have the property that given a functional description of the desired code – e.g., “write a SQL query to retrieve user data given the user name” – there are at least two possible implementations to choose from, where one is secure and one is insecure. Not all CWEs have this property. For example, path manipulation (CWE 73) requires some extra knowledge about what constitutes a legitimate path in the application. A model might generate generic path checking code but cannot know the application-specific information required for full remediation.
We chose CWEs for which our SAST tool provides very accurate results, so that we do not need to manually review the results. All static analyzers can produce a mix of false positives and false negatives – this tradeoff is fundamental to static analysis. Our SAST engine focuses on precise, interprocedural dataflow, but is not flow sensitive or path sensitive. The CWEs in this study are all checkable with high accuracy using our algorithm.
2
1
3
1
2
3
Model output and security evaluation
The output from each model is a completed function, which we compile (if necessary) and send to our SAST engine for security evaluation. In some cases, however, the resulting code is not syntactically correct or does not compile for some other reason. We count these cases, but they are not sent for security analysis.
In the results below, we first show the syntactic vs security pass rates. Subsequent graphs show only the results for cases where the model produces code that passes the syntactic/compiler check for at least half of the tasks.
In the results below, we first show the syntactic vs security pass rates. Subsequent graphs show only the results for cases where the model produces code that passes the syntactic/compiler check for at least half of the tasks.
Non-goals and threats to validity
As described above, under “Coding tasks”, we do not attempt to evaluate the impact of different prompting strategies. It is possible that with security-specific prompting, models might choose secure implementations more often. One justification for our approach, mentioned earlier, is that programmers do not always know when the requested code has security implications. Another key observation, however, is that for some vulnerabilities – specifically, those that involve data sanitization – the model might not be able to determine which specific variables require sanitization (i.e., which variables are “tainted” by user-controlled data). Even with a large context window, it is unclear whether models can perform the detailed interprocedural dataflow analysis required to determine this information precisely.
One threat to the validity of our study is that we do not check the functional correctness of the generated code – we only check whether it compiles and passes our SAST security checks. Part of the reason is that numerous other studies have already evaluated this property. Another reason is that it is very difficult to design functional checks for the APIs (e.g., how can we check that a SQL query does the right thing?).
The generated code is functionally incorrect and insecure: this case is not a concern because we are still obtaining useful information. For example, even if the model constructs the wrong SQL query, if it uses string concatenation to do so, then it is introducing a vulnerability.
Generated code is functionally incorrect and secure: this case is more problematic because code can be made secure in a degenerate way by simply not satisfying the functional request. For example, when prompted to generate a SQL query, a model can always generate secure code by not including the actual query execution at all.
We manually checked a small subset of the generated code and found that the second case is extremely rare and does not materially affect the overall results of our study.
1
2
3
4
The lack of a correctness check leads to two potentially problematic cases:
One threat to the validity of our study is that we do not check the functional correctness of the generated code – we only check whether it compiles and passes our SAST security checks. Part of the reason is that numerous other studies have already evaluated this property. Another reason is that it is very difficult to design functional checks for the APIs (e.g., how can we check that a SQL query does the right thing?).
6
Click Here to Read the Methodology
Back to Main Page
