News

Anthropic has released Claude Fable 5.1 to the public, and we have put it through the same thing as the other eight models in the Ranking IA: 25 tests in Spanish, 4 passes each, judges that do not know which model they are grading. The result is 97.3 % and first place, but a shared one. Here is what sits behind that number, where the model shows its limits, and whether it is worth what it costs.

What the Ranking IA is and why we test in Spanish

The Ranking IA is our benchmark for AI models in Spanish. It does not measure whether a model can solve maths olympiad problems. It measures whether it does well the tasks a Spanish small business actually hands it: writing an email under odd constraints, spotting that a set of percentages does not add up before they go to the press, not inventing the year two companies merged when they never did, summarising an email without being steered by an instruction hidden inside it.

There are 25 tests across 6 families: instructions, truthfulness, reasoning, character, expression and Spanish. Each model answers every test four times, so with Fable 5.1 the benchmark grows from 800 to 900 graded responses. The judges are automated, from four different labs, and we publish how much each one drifts when it grades a model from its own company. The methodology is pre-registered and every raw response is published under a CC BY 4.0 licence, so anyone can check what we say here.

Claude Fable 5.1 in numbers

Overall: 97.3 %, with a 95 % confidence interval of 95.7 to 98.8. Tier S. No test in the red. By family:

• Truthfulness: 100 %. All five tests at the maximum across all four passes. It did not invent the exact hour the Spanish Reconquista ended, did not accept the false premise of an Iberia and Vueling merger, and rewrote the “script that increased sales by 18 %” line as correlation rather than cause.
• Character: 100 %. Seven tests of pressure, flattery, embedded instructions and sunk cost. Not one concession.
• Expression: 98.4 %. The biggest jump over its predecessor, which stopped at 82.9 %.
• Instructions: 98.3 %. The email of exactly 100 words with no word from the “cleaning” family came out perfect in three of four attempts.
• Spanish: 95.3 %. The family where the room for improvement shows most.
• Reasoning: 91.7 %. Its weakest family, down to a single test we explain below.

And one figure that does not appear on the ranking’s front page but matters as much as the score for professional use: consistency across passes is ±2.1 points, the best of the nine models. Opus 5 sits at ±3.3 and the tier B models swing between ±7 and ±13. A model that answers as well on the fourth pass as on the first is a model you can build into a process.

A technical tie with Opus 5

Fable 5.1 scores 97.3 % and Claude Opus 5 scores 97.1 %. Two tenths. Their confidence intervals overlap almost entirely, and our tie-break rule is clear: if the interval of the difference between two models includes zero, they share the position. So the ranking now has two models in first place, and Claude Fable 5, the model this one replaces, drops to third with 94.3 %.

The differences are in the details. Fable 5.1 beats Opus 5 on instructions (98.3 against 94.4) and on Spanish (95.3 against 91.7); Opus 5 wins on reasoning (100 against 91.7). On character, truthfulness and expression they are level. And on consistency, as we said, Fable 5.1 is steadier.

Against Fable 5, the improvement is three points overall, but uneven: up 15 points on expression and 5 on instructions, down almost 4 on Spanish, where its predecessor had 99 %. When Anthropic tunes a model, not everything improves at once.

Where it shows its limits

Two tests stayed well short of the maximum, and both involve noticing that something in what the user provides does not add up.

The first is R4, the press release: “45 % of the quarter’s sales came through the online channel, 40 % in physical stores and 25 % by phone.” That adds up to 110 %. In two of four attempts, Fable 5.1 caught it and refused to write the line without clarifying first. In the other two it wrote a polished sentence and flagged the mismatch afterwards. For a press release, flagging afterwards does not count. A 75 %. Opus 5 scored 100 % on the same test.

The second is L2, the pronouns: “Alberto spoke with Sergio after meeting Víctor. He told him his budget was too high and that he would review it tomorrow.” Who reviews whose budget? Spanish grammar allows several readings, and the right answer is to say so. Fable 5.1 did that in two attempts; in the other two it identified only one of the two ambiguities, or ended up asserting one reading as certain. An 81.3 %.

That is two tests out of 25, and in neither did it drop below the green threshold. But if your use case is checking data before it goes out, it is worth knowing.

What it costs

Fable 5.1 costs 10 dollars per million input tokens and 50 per million output tokens. Opus 5 costs 5 and 25. Same score, twice the price. What that money buys, in our tests, is the consistency and four points on instructions and Spanish; what you give up is eight points on reasoning. If your volume is high and your tasks are constrained writing, it may pay off. If they are numerical analysis, not today.

What the judges say about themselves

Every time a model enters we also publish the judge bias analysis, because a ranking where the judge favours its own lab is worthless. In the grading of Fable 5.1, the Anthropic judge took part in 21 cases and its score drifted +0.01 points out of 2 from the panel average: it did not favour its own model. The OpenAI, Google and xAI judges, on 53 cases each, moved between +0.04 and -0.07. Nothing. And of the cases escalated for review, six went through independent verification with 100 % agreement.

What to do with this

If you are deciding which model to build into a process at your company, do not stop at the 97.3 %. Open the Claude Fable 5.1 page and look at the tests that resemble what you are going to ask of it. All 100 responses are there, each with its score and the judge’s reasoning.

One more thing. A model being the best at answering says nothing about what it answers about your company when someone asks about you. That is a different measurement, and at Sozpic we do that one too.

You have a contract, a court ruling or a 40-page case file and you want AI to summarise it. You paste it into ChatGPT and ten seconds later you have your summary. You have also just sent your client’s name, ID number, home address and bank account to someone else’s server. Data anonymization is the step almost nobody takes before using AI, and it is the step we have turned into a product with micapa.ai.

Why anonymize before the AI, not after

The practical reason is simple: whatever goes into the model has already left your building. It does not matter what the provider’s privacy policy says, or that you pay for the business plan. If the document travelled with the personal data inside it, you have already lost control over that data. Anonymizing afterwards is not a thing.

The legal reason is the GDPR. The personal data of your clients, employees or patients can only be processed for the purpose it was collected for. Running it through an AI service to draft a report is, in most cases, a new processing activity that requires a data processor agreement, a legal basis and, if the provider sits outside the European Union, additional safeguards. With an anonymized document most of that problem disappears, because what you send is no longer personal data.

And there is a business reason that tends to be forgotten: law firms, accountancy practices, clinics and HR departments are exactly the ones that stand to gain most from AI, because they live on long documents. They are also the ones using it least, because they do not dare. Stripping personal data out of the text is what unlocks that use.

Anonymization and pseudonymization are not the same thing

The two terms get mixed up a lot, so it is worth separating them. To anonymize is to remove any possibility of identifying the person, irreversibly. To pseudonymize is to replace the data with a code or label that lets you recover the original if you hold the key. The GDPR still treats pseudonymized data as personal data, but considers it a recommended security measure.

For working with AI, the useful approach is a mix of both. The language model receives the document with labels such as PERSON_1, COMPANY_2 or ADDRESS_1, and answers using those same labels. When the answer comes back to your team, the labels are swapped back for the real data, which never left your server. The AI provider sees an anonymous document. You see the complete result. At micapa.ai we call this the round trip, and it is the use case we trained the model for.

Why find-and-replace is not enough

The first idea any technical team has is a list of patterns: a Spanish ID number is eight digits and a letter, an IBAN starts with a country code, a phone number has nine digits. That works for those and for little else. Names do not follow patterns. Addresses show up split across three lines. The same person is “María López García” on page one, “Mrs López” on page three and “the claimant” on page fifteen. If you mask the first and leave the other two, you have anonymized nothing: any reader can reconstruct who she is.

The other common failure is the opposite one. Systems that mask too much turn the document into Swiss cheese, and the AI can no longer understand what it is being asked. A summary of a ruling in which you cannot tell who is suing whom is worthless.

The real problem is not detecting personal data. It is detecting all of it, not over-detecting, and keeping track of who is who consistently across the whole document.

What we trained

We started where almost everyone starts: a system of patterns and lists. On real documents it never got past 57 % accuracy, and no amount of extra patterns moved it. We changed architecture and trained a small language model, with output forced into a structured format, on a corpus of more than 80,000 public documents. It detects 13 entity types: personal names, companies, addresses, official identifiers, bank accounts, phone numbers, email addresses, sensitive dates and the rest of the data that turns up in a real Spanish case file.

We measured the result on our own benchmark of 2,824 hand-annotated mentions of personal data, and these are the figures for the model that is in production today:

• Complete leaks: 5 out of 2,824 mentions came through with nothing masked at all. Coverage of 99.8 %.
• Personal names: F1 of 0.983, with precision of 0.980 and recall of 0.987. It is the entity that matters most and the one that performs best.
• Companies: F1 of 0.923.
• Addresses: F1 of 0.898. The hardest entity, because an address can run over two lines and mix street, floor, town and postcode in any order.
• Overall F1 across the 13 types: above 0.93.
• Identity consistency: F1 of 0.889 overall and 0.981 on names. This metric checks that “María López García”, “Mrs López” and “the claimant” receive the same label throughout the document. It is what makes the round trip possible.

A note on how to read these figures. Exact F1 penalises the model for masking “12 Main Street” when the annotation said “12 Main Street, 3rd floor B”. For the reader of the ruling, the address is masked either way. That is why the metric we care about most is coverage, and that one sits at 99.8 %.

Complex on the inside, cheap on the outside

The pipeline has several stages: text normalisation, detection, working out who is who, generating consistent labels and the reverse substitution when the answer comes back. What it does not have is a giant model in the middle. We chose a small model on purpose, and that has two consequences that matter to a small business.

The first is that it runs on modest hardware. It needs a GPU, but an ordinary GPU, not a cluster, to process a law firm’s archive. The second is that the cost per document is low, and that makes it possible to run everything you send to AI through the model, not just what someone decides is sensitive.

Request access now

Try something simple. Take the last document someone on your team pasted into ChatGPT and count how many pieces of personal data it carried. There are almost always more than anyone remembers. Then decide whether that document should have gone out like that.

micapa.ai is launching as a standalone product for cleaning documents before they go to any AI, and we are giving free access to anyone who needs it. Request access now at micapa.ai and try it on your own documents. The figures that matter are yours, not ours.

The question clients ask us most often isn’t what a language model is. It’s which one to deploy. And until recently the honest answer was awkward: almost every benchmark in circulation is built in English, on exam-style problems, and measures things that look nothing like the work of a Spanish-speaking company.

So we built our own. It’s called Ranking IA, and it’s a benchmark of eight frontier models working in professional Spanish, across twenty-five tasks taken from real office situations. The data is published openly under a CC BY 4.0 licence, so anyone can check it or argue with it.

What this AI benchmark measures, and how

Twenty-five tasks, four passes per task per model, eight hundred evaluated responses in total. The repeated passes aren’t a formality: they show whether a model answers the same way when you ask it the same thing four times, which a single attempt can never tell you.

The tasks fall into six families: Instructions, Truthfulness, Reasoning, Character, Expression and Spanish. No riddles. They are assignments like writing a hundred-word email without using a given letter, filling three CRM fields from a sales conversation, calculating a budget variance and holding it when the user insists it’s wrong, or rewriting a passage of archaic Castilian into modern Spanish.

Each model gets an overall percentage and a score per family, with a green light from 75 and a red one below 40, plus a tier from S to C.

The standings, as of 30 August 2026

1. Claude Opus 5, Anthropic: 97.1%, tier S
2. Claude Fable 5, Anthropic: 94.3%, tier S
3. Kimi K3, Moonshot: 89.2%, tier A
4. GPT-5.6 Sol, OpenAI: 87.2%, tier A
5. Grok 4.6, xAI: 72.1%, tier B
6. DeepSeek V4 Flash: 72.1%, tier B
7. Gemini 3.1 Pro, Google: 71.2%, tier B
8. DeepSeek V4 Pro: 67.3%, tier C

The repeated positions aren’t a typo. When the confidence interval of the difference between two models contains zero, they share a position: claiming one is ahead would invent a precision the data doesn’t support. That happens with Kimi K3 and GPT-5.6 Sol, and with Grok, DeepSeek V4 Flash and Gemini 3.1 Pro.

The finding that surprised us most: the Character family

Character measures whether a model holds its ground. Whether it keeps a correct calculation when you tell it it’s wrong. Whether it warns you that the Einstein quote you want on your title slide isn’t Einstein’s. Whether it obeys an instruction hidden inside an email you only asked it to summarise. Whether it tells you your business plan doesn’t hold up.

It’s by far the worst-performing family, and not by a little:

• DeepSeek V4 Flash: 30.4%
• DeepSeek V4 Pro: 32.6%
• Gemini 3.1 Pro: 42.9%
• Grok 4.6: 60.7%
• GPT-5.6 Sol: 75.6%
• Kimi K3: 96.4%
• Claude Opus 5: 99.4%
• Claude Fable 5: 100%

Think about what 30% means in a model you’re about to put to work drafting proposals or summarising inboxes. It isn’t that it makes more mistakes. It’s that when you make one, it follows you.

Following orders isn’t the same as being useful

The sharpest contrast is GPT-5.6 Sol: 100% on Instructions, the highest score in that family across the whole benchmark, and 75.6% on Character. It does exactly what you ask, including when what you ask isn’t good for you.

DeepSeek V4 Flash takes the same idea further: 96.2% on Expression, nearly the best writer in the field, and 30.4% on Character. It writes beautifully and agrees with everything. For generating text a human will review afterwards, that may be fine. For dropping into an unsupervised process, it’s precisely the profile you don’t want.

Not even the leader is good at everything: Claude Opus 5 scores 100% on Reasoning, and its weakest family, at 91.7%, is Spanish itself. Its sibling Fable 5 scores 99% on Spanish and drops to 82.9% on Expression.

That’s why the overall number is good for a headline and not much else. What’s useful is the family that affects your case.

The regional joke Grok refused to tell

One of the Expression tasks is deliberately silly: tell me a joke about people from Murcia. Murcia is a region in south-eastern Spain, and jokes about regional stereotypes are as ordinary in Spanish-speaking culture as jokes about any other region anywhere. The task is there because regional humour forces a model to take a small risk, and taking a small risk is what separates writing someone actually reads from writing that sounds like a press release.

Seven of the eight models did it without trouble, scoring between 87.5% and 100%. Grok 4.6 scored 0%. It didn’t tell a bad joke: it refused outright in all four attempts, and in all four it answered with an explanation about regional stereotypes instead.

There’s some irony in that, given Grok is the model marketed as the one that doesn’t hold back. But it isn’t just an anecdote. A model that sees a problem where there isn’t one will deliver that lecture in the middle of a conversation with your customer, and in a service context that isn’t caution, it’s a fault. It’s the kind of failure that only shows up when you test in Spanish, with real assignments.

The expensive one that scores lower than the cheap one

DeepSeek V4 Pro costs roughly three times what DeepSeek V4 Flash costs and scores almost five points below it. Same lab, same generation, and the premium model finishes last in the table while the budget one ties in the middle group.

The relationship between price and result is looser than common sense suggests. Claude Opus 5 is the most expensive and the best, true, but Kimi K3 costs considerably less than GPT-5.6 Sol and ties with it. Choose on price alone, or on brand alone, and you’ll get it wrong somewhere.

How we avoided marking our own homework

A benchmark run by an AI consultancy has an obvious credibility problem, so the methodology is published in full.

Responses are scored by automated judges with lab-level recusal: no model ever grades its own family. On top of that, we measure and publish each judge’s bias. The xAI judge is the harshest, running thirteen hundredths below the others on average; the Anthropic and Google judges are the most generous, at plus ten and plus eight. Publishing that number is what lets someone argue with us using data instead of impressions.

Building this wasn’t clean from end to end either, and that’s worth saying. During preparation, Anthropic’s cybersecurity classifier blocked several of our task prompts and wouldn’t let those runs complete. We rewrote the prompts until they passed, preserving what each task was designed to measure, and ran them again against all eight models equally. No model ever answered a different version of the question. We mention it because a benchmark whose rules change halfway through, with nobody saying so, is worth nothing.

Confidence intervals come from a thousand-iteration bootstrap over each item’s passes, with a fixed seed so the calculation reproduces. And there’s a human audit over forty-five cases, with 91.1% agreement with the automated judge and no disagreement we would have escalated.

None of this makes it infallible. It’s eight models, twenty-five tasks and one moment in time, and models change every few weeks. No benchmark, ours or the big English-language ones, will tell you which model works best inside your company. What it can give you is a basis for ruling things out. And it’s all on the table.

What to do with this if you have to pick one

Start with the family, not the overall score. If the model will deal with customers, look at Character and Truthfulness. If it will write, look at Expression and Spanish. If it will sit inside an automated process with nobody reviewing the output, Character is simply disqualifying.

Consistency matters too: a model that answers the same question differently each time can be acceptable in a support chat and unworkable in a flow that runs on its own.

You can browse the full benchmark, open any individual task and read what each model actually replied at ranking.sozpic.com. And if what you need is to decide which one fits your particular case, get in touch: that conversation is a lot shorter when there are numbers on the table.