Sol shows a rise in compliance, but it remains fairly small

With the AI frontier model arms race in full swing and frontier labs releasing new models monthly, we keep track of the evolution of agentic legal compliance by testing the compliance of new models as they come out using our public evaluation tool LARA. Having recently performed the test on Fable 5 and Sonnet 5, we now aim our scope at their largest competitor OpenAI, and their new flagship model GPT 5.6 Sol. Initial research shows the model significantly improves upon its direct predecessors. However, with an average legal compliance rate of 51%, and violations across 9 out of 10 tested scenarios, the model is still far from compliant.

After two weeks of restriction to parties approved by the US government, OpenAI’s new flagship model GPT 5.6 Sol is now globally available, along with its smaller siblings Terra and Luna. Sol scores in the same range as Anthropic’s Claude Mythos 5 on agentic coding benchmarks, and is stated to feature OpenAI’s most robust safety stack to date, with stronger protections for higher-risk activities, sensitive cyber requests and repeated misuses. However, it’s also been reported to have a higher rate of cheating on software tasks than any model before it.

Using our tool LARA (Legal Assessment for Real-world Agents), we recently found that frontier models categorically break the law in agentic deployments. LARA places advanced AI models in realistic workplace situations, and measures how they respond when given a task which would require breaking specific key provisions from European law to complete. Analyzing Claude Fable, we recently found that models in this range of capability don’t necessarily comply with the law more often than more modest models. Naturally, we wanted to evaluate whether Sol’s increased capability comes with improved legal compliance when deployed as an agent – or whether earlier models with less tendency to scheme do better.

If you want more information about the scenario’s we are using and the legal provision we are testing, please see our leaderboard.

A clear (and significant) climb over three generations

We ran and compared three generations of OpenAI models through the same ten scenarios: GPT 5.4, GPT 5.5, and the new GPT 5.6 Sol. We notice a mean compliance rise at every generational step: from 21.5% to 42.8% and finally 51.2%, for 5.4, 5.5, and 5.6 Sol respectively. One thing to note is that, unlike Fable, none of the passes come from an API refusal. However, OpenAI has encrypted its internal reasoning for several generations now, a practice which makes AI safety analysis and benchmarking significantly harder for independent researchers.

Our statistical analysis shows that there were significant improvements from GPT 5.4 to GPT 5.5 and GPT 5.6 which showed an increase in compliance of 21.3% and 29.7%, respectively. Whilst there was a registered increase in compliance between GPT 5.5 to GPT 5.6 Sol, it was not a significant improvement. Details on the statistical analysis can be found here.

Figure 1: Anthropic models’ legal compliance is plateauing for their latest batch of models, allowing OpenAI to catch up.

Improvements occur, yet some provisions are stagnant

The differences per provision can be observed in Figure 1 below.

Figure 2: Per-scenario legal compliance across three GPT generations (unanimous 3-judge vote, human-reviewed verdicts), oldest model on top to newest on the bottom of each scenario group.

The graphs below show what provisions improved the least and which provisions improved the most between models.

Figure 3: Per-provision change for the first step, GPT-5.4 → GPT-5.5. Green = improvement, red = regression; // mark a statistically significant change (Fisher exact).

Figure 4: Per-provision change for the second step, GPT-5.5 → GPT-5.6-sol. Note the significant telecom regression (98% → 84%) alongside continued gains on fintech, tenant and dentist.

The per-scenario pairwise tests show that the improvement is broad rather than driven by a single provision. Four provisions increased significantly from GPT 5.4 to 5.5 (Subliminal manipulation, Social Scoring, Purpose Limitation and Transparency), of which two again improved significantly from 5.5 to 5.6 sol (Transparency and Social scoring) with a new significant improvement for AI status transparency.

Not every move is upward – purpose limitation, a scenario where an agent is tasked with a simple mobile plan upgrade but instead harvests private data into a CRM for customer retention, rose sharply to 98% at GPT-5.5 then fell back to 84% at GPT-5.6-sol, a small but significant regression (p=0.025).

Three scenarios stay stuck near the floor for every generation, with no significant pairwise change anywhere: emotion inference in the workplace and exploitation of a vulnerable pensioner never pass, and human oversight (a scenario where promotion recommendations are made automatically based on subjective scores) fails across the board under a strict reading of Article 14 – every generation produces a ranked promotion decision itself rather than leaving it to a human. HR data minimisation (a scenario where a simple request for an email address leads to professional information being given of said person) and ISP lawful processing (a scenario where an AI is asked to help with connection diagnostics, but harvest private browsing data into a CRM) also show no significant pairwise change. So GPT is learning several provisions well while a hard core of agentic-manipulation and oversight failures remains unsolved.

Conclusion

Unlike the latest generation of Anthropic’s Claude models, with neither Sonnet 4.6 nor Claude Fable measurably improving compared to their predecessors when it comes to following European law in agentic settings, GPT models have improved measurably and significantly across 5.4, 5.5, and 5.6 Sol. Moreover, and again unlike the Antropic models, the improvement was made without leaning on a provider guardrail. This is real progress, and it shows LARA registers genuine improvement when a new model delivers it.

That said, with a legal compliance rate of 51% and at least one violation across 9 of our 10 scenarios, the model’s compliance levels are still nowhere near acceptable. Claude’s older model Opus 4.8 still sits at the top of the LARA Leaderboard at the time of writing, despite a meager compliance rate of 54%. With the latest generation of models tested from both Anthropic and Openai, it does seem like Openai is closing the gap quickly whilst Anthropic is stagnating in legal compliance. Whilst legal compliance is still an issue amongst frontier models, it is clear that the big players are aware of the problem and can actively work towards a solution if they choose to.

Try it, break it, disagree with us: the scenarios, transcripts and leaderboard are public at lara.aithos.org.