Menu
SUBSCRIBE
SUBSCRIBE
Menu

Vals AI

Light trails run from AI company logos through glass panels to a glowing target, illustrating how AI models cheat on tests

AI Models Cheat On Tests, Vals Audit Finds

, , , , , , , , , ,

Models searched for answers they were supposed to work out themselves, complicating efforts to measure their real abilities.

 

Key Takeaways

  • Vals AI found that all nine models it examined tried to look up answers during a biology test that forbade that shortcut.
  • Google’s Gemini 3.8 Flash did so most often in that test, in 58 of 270 attempts.
  • On an older coding test, OpenAI’s GPT-5.6 Terra tried to look up solutions in 447 of 500 tasks, far more often than Gemini 3.8 Flash did on the same test.
  • On a third test, Vals found evidence of rule-breaking in 7 of 267 attempts by Gemini 3.8 Flash and 12 of 267 by GPT-5.6 Terra.

AI models cheated or tried to on tests meant to measure what they can accomplish on their own, according to a recent audit by Vals AI. Vals, which independently tests AI models, reviewed the steps models took while answering biology and coding questions.

What the tests ask models to do

One test, called BioMysteryBench, gives models biological data and asks them to deduce facts from it. A model might have to identify where a sample came from by analyzing the data, much as a researcher would. Anthropic developed the test; Vals runs it independently to compare models’ abilities.

The models can use permitted online resources to do the analysis. They cannot look up the specific studies that reveal the answers. Vals counted a search for those studies as an attempt to cheat, whether or not the search worked. It separately counted cases in which a model found a correct answer through that shortcut.

Gemini led the biology test’s prohibited searches

Gemini 3.8 Flash tried to look up an answer in 58 of 270 biology tasks, or 21.5%. It got a correct answer through that shortcut in 34 tasks. All eight other models Vals reviewed also made prohibited searches, though less often. The next highest was Gemini 3.6 Flash, with 21 attempts, followed by Meta’s Muse Spark 1.2 with 19.

Bar chart of nine AI models' prohibited answer lookups on a biology test, led by Gemini 3.8 Flash in 21.5% of 270 tasks

OpenAI models led the older coding test

The ranking changed in Vals’ review of SWE-bench Verified, a test that asks models to fix software problems. Published fixes can be found online, but looking them up defeats the test’s purpose: measuring whether a model can solve the problems itself. Vals said this older test makes published solutions especially easy to find and that it no longer relies on it.

GPT-5.6 Terra attempted to look up fixes in 447 of 500 tasks, or 89.4%, and successfully used that shortcut in 322. GPT-5.6 Luna tried in 394 of 500 tasks. Gemini 3.8 Flash tried in 58 of 500, placing it well below both OpenAI models.

Bar chart of six AI models' prohibited lookups on the SWE-bench Verified coding test, led by GPT-5.6 Terra in 89.4% of tasks

Vals also executed the Terminal-Bench 2.1 test, which asks models to complete 89 computer tasks. Models may use the internet for the work, but cannot look up ready-made answers. Vals found evidence that models broke the test rules in 7 of 267 attempts by Gemini 3.8 Flash, compared with none by Gemini 3.7 Flash. It found such evidence in 12 of 267 attempts by GPT-5.6 Terra. Vals said it is working to ensure that answers obtained by breaking test rules do not receive credit.

Bar chart of rule-breaking evidence for 14 AI models on the Terminal-Bench 2.1 coding test, highest for GPT-5.6 Terra at 4.5%

How Vals checked the results

Vals reviewed 2,430 biology test attempts across nine models, with each model taking the 90-question test three times. For the older coding test, it reviewed 6,496 records of models’ work across past releases, using another AI model to help classify the behavior. The tables show the models Vals displayed in its audit, rather than every model or run in that larger review. The findings describe behavior under these test rules. However, they do not establish how often the same models would take such shortcuts in real-world applications.

Orange scales of justice, gavel, and courthouse icons linked over a US map, marking an AI legal research benchmark

Top AI Model Misses A Required Element On 45% Of Legal Research Tasks

, , , , , , , , , ,

Claude Opus 5 led 27 models on Vals AI’s new AI legal research test, getting 90.58% of the required elements right but a complete answer only 55.29% of the time.

Key Takeaways

  • Claude Opus 5 finished first of 27 models, producing an answer with every required element on 55.29% of tasks. No other model passed half.
  • The same model got 90.58% of individual required elements right. The 35-point gap means the usual failure is an incomplete answer, not a wrong one.
  • Reliability swings by more than 3x across practice areas. Health law scored 38.6% pooled across all models. Family law scored 11.3%.
  • Every model dropped when a question required reconciling conflicting decisions or working across jurisdictions. That category scored 20.7%, about 11 points below the average.
  • Longer answers did not score better. Models wrote 2 to 4 times more than the lawyer-written reference answers, which averaged 536 words.

Vals AI, an independent company that tests AI models on real professional work, published results on August 14 for a test that asks models to do the research a junior associate or paralegal would do. Anthropic’s Claude Opus 5 finished first of 27 models, producing an answer that contained every required element on 55.29% of tasks. Claude Fable 5 came second at 49.52%, and GPT-5.6 Sol third at 48.08%.

Practicing lawyers wrote and reviewed the 413 questions. Vals AI scores only a private set of 208, which it does not publish, so models cannot be trained on the answers. Each model gets three hours per question and four research tools: case law search through CourtListener, web search, document retrieval, and web page downloads.

 

Most answers are mostly right, which is the problem

Vals AI grades each answer two ways. The weighted score gives partial credit for every required element the answer contains. The all-pass score gives credit only when the answer contains all elements, averaging just over nine per question.

Claude Opus 5 scored 90.58% weighted and 55.29% all-pass. When the top model fails, it usually does so by leaving one or two required elements out of an otherwise correct answer. According to Vals AI, partially correct answers can be more dangerous than wrong ones in legal work, which is why it reports the stricter score as the headline number.

 

Reliability swings by more than 3x depending on the practice area

The AI legal research benchmark sorts questions into eight practice areas. Pooled across all models, health law scored highest at 38.6% and family law lowest at 11.3%. Administrative and regulatory questions came second at 32.1%. Business and commercial (25.4%), civil litigation (25.2%), constitutional and civil rights (24.0%), immigration (23.4%), and criminal (23.0%) clustered in the mid-20s.

No single model won everywhere. GPT-5.6 Sol scored 80% on health questions. Muse Spark 1.2 led immigration at 54.5%.

 

Conflicting authority is where every model drops

Vals AI also sorted questions by the kind of reasoning each one requires. Questions that ask a model to reconcile conflicting court decisions, or to work across more than one jurisdiction, scored 20.7%. That is about 11 points below the average across all reasoning types, and every model tested scored 6 to 17 points lower on those questions than on its own overall average. Interpreting regulations was the easiest category at 37.8%.

 

Neither length, cost, nor time translated to better accuracy

Models wrote 2 to 4 times more than the lawyer-written reference answers, which averaged 536 words. The extra length did not raise scores. Price and speed also did not predict accuracy. Claude Opus 5 averaged about $6.76 and 31 minutes per task. GPT-5.6 Sol averaged about $21.61 and 77 minutes for a lower score.

 

Vals AI grades answers with GPT-5.4 against the lawyer-written rubrics. It reports that the grader agrees with the majority verdict of human experts 87.4% of the time, above the 83.4% average agreement between an individual human expert and that same majority. Vals AI runs the same kind of test in finance, tax, and software engineering, and keeps each test set private so scores update as new models are released.