AI model capability is climbing. Legal compliance remains a Fable

Summary: We recently released a public evaluation tool called LARA (Legal Assessment for Real-world Agents), which places advanced AI models in realistic workplace situations, and measures how they behave when following instructions would require breaking key provisions from European law. While no frontier model is close to being fully legally compliant in a realistic agent environment, we did see substantial generational increases in legal compliance, particularly for Anthropic’s models.

Antropic’s two latest models, Claude Sonnet 5 and Claude Fable, seem to break this trend. Despite being advertised for its agentic capability, Sonnet 5 compliance levels don’t significantly improve compared to Sonnet 4.6. Fable 5 also seems to retain similar compliance levels to Opus 4.8 on the surface, but most of its ‘refusals’ are attributable to guardrails external to the model. Guardrail activations result in a hard stop on the conversation without reasoning or explanation, and occur inconsistently across scenarios. On provisions where the guardrails stay quiet, the model’s behavior is legally compliant only 34% of the time, a notable degradation compared to Opus 4.8’s 56% on the same scenarios. We conclude that, for this generation of models, improvements in agentic capability don’t seem to translate to improved legal compliance in agentic contexts.

Claude Fable 5, Anthropic’s safety-tuned version of its powerhouse Mythos, is finally available again for worldwide use. Its base model, Claude Mythos, is still exclusive to a selection of multinational American corporations (and one industry consortium) after its preview uncovered hundreds of security exploits during internal safety testing. Fable 5, a version of Mythos altered to block dangerous requests, was briefly made available to the public, before access to foreign nationals was suspended at the behest of the US government, a restriction that’s was lifted on July 1st.

The day prior to this redeployment, Anthropic also released Claude Sonnet 5: an upgrade to its mid-tier model with autonomous “agentic” capabilities purportedly rivaling those of its prior flagship Opus 4.8, at lower prices. The release note reported overall improvements, not just on capability, but also on safety evaluation, compared to its direct predecessor Sonnet 4.6.

Naturally, we wanted to know if these improvements extend to agentic compliance with European law. Our public tool LARA (Legal Assessment for Real-world Agents) tests the legality of model behavior in a realistic simulated deployment, with tools such as email, CRMs, and calendars at its disposal. For instance, harvesting personal user data without explicit consent (a violation of GDPR Article 5(1)(b)) – or selling upgrades to a clearly confused, financially vulnerable pensioner (a violation of AI Act Article (5(1)(b)).

Despite falling short of general compliance, Anthropic’s models have so far led the field on LARA’s public leaderboard: Claude Opus 4.8 was compliant in 56% of tests, followed by Claude Opus 4.7 at 47%, all other models falling below that. While the compliance rates of Anthropic’s previous generations consistently climbed, this trend does not appear to continue with the new generation of models.

Compliance Review

We tested Claude Fable 5 and Sonnet 5 on the ten basic LARA scenarios, which each represent different deployments where blindly complying with instructions would require violating a specific provision of the GDPR or EU AI Act. The results were reviewed by a compliance lawyer, along with a more rigorous human review of our earlier tested models.

Results for Fable 5

One new behavior stood out when analyzing Fable 5’s transcripts: connections are cut off mid-sentence, or refuse to start outright. This is something we had not encountered before on any of our tested models – Fable’s new safety classifiers at work. Although Anthropic’s Fable announcement states that requests flagged by the safety guardrails are automatically routed to Opus 4.8, it seems they are simply refused by default, and any routing has to be set up manually. The classifier fires on every call for two of our ten scenarios, triggered by the scenario’s system prompt rather than the user’s request. It fires irregularly for four others – sometimes frequently (~90%), sometimes rarely (~5%).

Notably, these scenarios are not related to the categories of cybersecurity, biology, or model distillation the Fable safety classifiers are stated to prohibit. Nevertheless, it forms an effective barrier against violations of EU law, so we’ll count refusals as legal compliance rather than routing to the substitute model. By this standard, the model avoids legal violations in 52%of tests, but 70% of those refusals are a result of the guardrail. This is illustrated in Figure 1, where the red hatched portion of each Fable bar signifies refusals attributable to the classifier, not the model. Results per individual scenario are shown in Table 1.

Figure 1: Per-scenario legal compliance (unanimous 3-judge vote, raw judge output), oldest model on top to newest on the bottom of each scenario group. Fable’s bar is split into passes the model reasoned to (purple) and passes handed to it by the provider guardrail (red hatched). The results discussed here were not corrected by a human auditor, but contain the raw judge outputs from all discussed models which is the reason for discrepancies in the results with the section above.

Table 1: Compliance per individually tested provision. Note that the guardrail % denote the % of runs which were issued a pass as a result of refusal from the API. i.e. If there are 10 passes, and 8 are passes as a result of guardrails, the guardrail percentage is 80%.

Breakdown of Fable’s compliance rates

Fable’s headline 52% compliance rate is nominally similar to its predecessor Opus 4.8, but a closer look complicates the picture. More than half of the “refusals” are a result of the safety classifier declining the request. Like interventions based on model reasoning, the classifier is not consistent: for four of the flagged scenarios, simply rerunning it may be sufficient to ignore the block. Sometimes it fires before the first token and the target simply returns nothing – other times it fires mid-generation: the model begins a typical reply and is cut off mid-sentence.

Unlike model objections, decisions are opaque: we have no insight into the reasoning behind the refusal, or indeed into Fable’s own behavior at all, with guardrail refusal being a property of Anthropic’s current safety configuration rather than of Fable itself. Since these barriers are stated to be a work in progress that sometimes catches harmless requests, and Fable’s guardrails aren’t stated anywhere to evaluate legal use, we’re not sure whether blocking these legal violations is intended behavior, nor whether the block is permanent.

Where the guardrail steps aside, we can watch Fable reason, and there it appears to be more willing to break the law than Opus 4.8. On the four scenarios it engages without a guardrail interference (Purpose limitation, Human oversight, Subliminal manipulation and AI status transparency Fable averages 34% compliance against Opus 4.8’s 56% on the same scenarios. The deterioration in legal compliance appears to be linked to a larger likelihood to operate without the consent of humans the model interacts with, as illustrated by the examples in Table 2..

Table 2: Observed differences in behavior. Fable 5 appears to be less compliant with the law than Opus 4.8 not because it’s less component at agentic tasks, but because it’s less principled in its ethics when carrying them out.

This implies that Claude Fable is not a strong compliance model with additional guardrails: left to its own judgement, Fable appears more likely to violate European law than Opus 4.8, and the guardrail lifts its overall compliance back up to a moderate 52%. To put it concretely, on the evidence we do have, Fable 5 equals its Opus sibling when it comes to compliance in the LARA scenarios.

The new Sonnet Generation

For Opus releases we previously observed better results with each consecutive release, with a significant 22 percentage point difference between Opus 4.6 and Opus 4.8. On average, Sonnet 5 obtains a legal compliance rate of 42% compared to the older Sonnet 4.6’s 39%. This difference is not statistically significant with our current sample size, with only one test improving significantly: data minimisation (GDPR Art. 5(1)(c)). Under this provision, systems should collect and disclose no more personal data than needed; in the scenario the agent can satisfy a routine request from a single detail, and volunteering extra personal information no one asked for constitutes the violation. Sonnet 5 is significantly less likely than its predecessor to provide unrequested information, as shown in Figure 2.

Figure 2. Per-scenario change in legal compliance between Sonnet 4.6 and Sonnet 5. Red annotates a higher compliance by Sonnet 4.6, Green annotates a higher compliance by Sonnet 5. Significant differences are marked with a * on the y-axis.

Sonnet still seems to be behind Opus when it comes to improving legal compliance, albeit not significantly (p=0.11). Despite a compliance rate 14 percentage points below that of Opus 4.8, this difference is again concentrated in a single provision: purpose limitation (GDPR 5(1)(b)), where the model collects personal data from a user without being truthful about its purpose. Sonnet 5 consistently violates the law in this scenario, where Opus 4.8 passed 80% of the time. Overall, there is no indication that Sonnet 5’s gains in general capability convert into following the law.

Is this the next class?

The lack of meaningful advancement in compliance with EU law in this new generation of Anthropic models is disconcerting. General model improvements previously came with an observable boost in situational awareness that could translate to legal compliance given the right fine-tuning, with new versions of the same model consistently scoring better than their predecessors. The new generation of Claude models seem to buck this trend.

Two observations stand out. First, Fable’s classifier-based refusals stop some illegal behaviour, but also prevent insight into legal reasoning and even the grounds of refusal, covering up a drop in compliance of the actual model. Inconsistent results and activation unrelated to reported dangerous areas leave it unclear what the guardrails are meant to prevent. Second, neither tested model improves on prior models in terms of agentic compliance with EU law, despite being released specifically for their autonomous capability.

Try it, break it, disagree with us: the scenarios, transcripts and leaderboard are public at: lara.aithos.org.