|
4 June 2026 (Washington, DC) - Our industry colleague, Doug Austin, had an interesting piece this morning on why, when it comes to large language models (LLMs), the probability that hallucinations will go away is zero. As Doug notes:
People think of LLMs (like ChatGPT, Claude, and Gemini) as knowledge bases, but they are far from that. LLMs are designed to predict language, not verify truth. That distinction is very important in understanding why these systems sometimes produce hallucinations, where the responses that sound convincing but contain inaccurate, misleading, or completely fabricated information.
And you can’t blame the data. The primary factor is that the model does not truly “understand” information the way humans do. Humans rely on reasoning, experience, and external validation to determine whether something is true. LLMs don’t possess that kind of grounded understanding. Instead, they rely on statistical relationships between words and concepts. If those statistical relationships suggest that a certain response fits the prompt, the model may generate it even when the information is wrong, because – rather than refusing to answer – the model may attempt to fill the gaps with language that appears reasonable.
The people he spoke with echo the same thoughts as our AI industry contacts: hallucinations are here to say. Doug's piece is very good, and well worth your time, and you can read it by clicking here.
And it is getting worse. Lawyers are increasingly naming specific legal technology artificial intelligence tools that they allege were involved in hallucination errors in court, pushing the legal tech industry into the spotlight.
For now, there isn't a reputational risk to the industry's top players over being called out in court filings. However, if enough instances of hallucinations are attributed to a vendor, rightfully or not, that could undermine their tool's standing over time.
In April, an attorney in the Eastern District of California blamed the AI legal research platform OpenCase for providing false citations in an opposition brief. In this incident, the attorney claimed the hallucinations occurred after using the tool to check citations.
Unlike past instances of AI tools writing briefs that were later found to include hallucinations, the attorney here claimed the tool changed the citations when he used it to refine the draft.
When our media partner Law360 asked OpenCase for a comment, the company did not respond.
As we have reported at length, past instances of hallucinations with legal AI tools have typically involved drafting. Today, hallucinations are springing up in research tools that have legal citations. Many of these legal tech providers use case law to build their tools, and note that some attorneys might use multiple legal AI tools for matters and cherry-pick the vendors they cite when caught with a hallucination.
But lawyers blaming tools for hallucinations is now becoming more routine than exceptional.
Studies show that most attorneys are now using over 800 different legal AI tools. At the same time, there have been over 1,300 cases as of May 2026 in which generative AI produced hallucinations in legal briefings, according to a database compiled by attorney and data scientist Damien Charlotin. If you subscribe to Doug's blog, he provides a running count every Friday on end-of-the-week blog.
Charlotin told Law360 that roughly 10% of the entries on his hallucination database include the name of the tool used. ChatGPT is the most commonly used tool behind those hallucinations, which Charlotin suspects is driven by pro se litigants, because counsel might be less willing to cite specific legal tools.
Most major legal AI vendors are listed at least once on the hallucinations database. A few of them get upset over these mentions, Charlotin said:
"I got emails low-key threatening me to remove the mentions. I doubt there would be any reputational impact, though, at least for the big ones".
He added that most major vendors no longer promise their tools will be free of hallucinations. Many have added new techniques that improve AI, such as retrieval-augmented generation, and legal citations, to lower hallucination rates, but emphasize the importance of human controls.
Thomson Reuters, which markets its AI assistant CoCounsel for legal work, published an article in August 2025 stating that attorneys are still responsible for checking facts before submitting a document to the court.
When lawyers have been caught submitting court documents with fabricated cases, some have apologized and vowed to stop using AI in the practice of law, while others have described the mistake as an "isolated event" and changed firm policies to prevent it from occurring again.
While it's possible that a legal AI tool could be liable for negligence, the facts in the usual scenario of hallucination make it highly unlikely that a lawyer or client could succeed in a legal action against an AI vendor, according to Daniel W. Linna Jr., director of law and technology initiatives at the Northwestern University Pritzker School of Law. Linna said the platforms warn users that AI tools can make errors, and most provide links to cases to help review citations:
"It does harm the reputation of AI providers when they are named in court orders, but savvy lawyers know that the errors say a lot more about the lawyer than the platform that they used".
Linna added that he is skeptical of some of the stories lawyers have told judges about errors with legal AI tools, because their conduct would look worse if they had used a consumer AI tool for research.
In March, a lawyer in the Southern District of New York attempted to attribute hallucinations to LexisNexis' Protégé, but LexisNexis later refuted that the lawyer had a subscription to its AI tools, according to court documents.
LexisNexis did not respond to a request for comment.
In the end, most of the egregious errors could be avoided if law firms used technology to check each filing before submitting briefs to the court, according to Linna. Vendor exposure doesn't relieve attorney accountability:
"Vendors are likely to develop reputations regarding the reliability and effectiveness of their AI tools, but those reputations don't change a lawyer's underlying ethical responsibility to ensure that the tools are used appropriately and that anything cited in a court submission is appropriately vetted".
Up to this point, courts have not sanctioned legal technology vendor for hallucination errors, but being named in a sanction against an attorney could potentially alter how legal AI is perceived and marketed. But one legal pundit told us:
"Up to now courts have exclusively sanctioned the attorneys and law firms who submit the hallucinated filings. Under long-standing legal principles like Federal Rule of Civil Procedure 11, attorneys have a strict, non-delegable duty of candor to the tribunal. This means the signing lawyer remains fully responsible for verifying every citation, regardless of whether they used a public chatbot or a premium, legal-specific AI vendor.
But while legal tech vendors have so far avoided courtroom sanctions, the landscape is shifting rapidly in terms of reputational fallout, legal blame shifting, and market liabilities. There are two cases I am watching that could raise market liabilities for legal tech vendors".
No, he did not reveal what cases but we'll stay in touch with him.
| 
