|
False citations are not uncommon in AI-generated legal briefs, which plague well-resourced BigLaw attorneys and pro se litigants alike. Courts have grappled with how to handle such problems, with standing orders that require AI disclosures and promises of human review, as well as sanctions for false citations.
It's easy to see why courts are alarmed.
And so legal media has been all over the Graciela Dela Torre case and the Nippon Insurance case. Those hallucinations present a threat to the legitimacy of legal argument, according to Brad Wendel, a Cornell law professor who writes about legal ethics:
"I'm really worried this is going to define the standard downward, and courts are going to start saying: 'Well, look, even Boies Schiller and Sullivan & Cromwell screwed this up. So, how can we punish a solo practitioner who's really busy or a pro se?' There has to be this really determined effort to hold the line and to keep the norm where it's at, and not let it erode. "
Another dilemma is posed by Dela Torre's alleged use of the chatbot for practical advice. She had settled her claims with Nippon in 2024, but a year later, worried its terms were the result of errors or omissions in the record, and wondered if she'd gotten a good enough deal. According to Nippon's lawsuit, when her attorney told her she couldn't reopen the case, she asked ChatGPT if he was gaslighting her, and the machine gave her an answer that confirmed her suspicions.
Chatbots are designed to please, and will often answer a leading question in the affirmative. That's not true of a human lawyer, who has a reputation, a law license, and time and resources to protect. AI may be able to spit out a passable legal argument - but it's no replacement for human judgment.
Angela Tripp, a program officer for technology at Legal Services Corporation, weighed in:
"Talk to a legal aid lawyer and they'll say — all day long, they tell people — 'No, you can't do that,' and disappoint people with the realistic shortcomings and limitations of our justice system. AI can't grapple with that. It wants to see that everything is possible, because probably somewhere in its database, there's some piece of information that says that it is possible, because it's feeding on what may or may not be true, or is only true in a particular situation".
According to Nippon's complaint, as I noted above, many of Dela Torre's filings predate an October 2025 addition to OpenAI's terms that prohibits users from turning to ChatGPT for legal advice. That provision will likely insulate it from liability involving future litigants, according to many legal experts.
But the case still poses ethics and liability questions for AI services, particularly if courts find the new terms don't provide them a meaningful shield. One lawyer on the Bloomberg Law panel noted:
"If that's not enough, is the response then that OpenAI and Google and Meta all need to reprogram their tool so that it will not give legal advice? I mean, do they need to reprogram it to not tell you how to fly a plane? I know they did reprogram it in the early days to not tell people how to build a bomb".
The lawsuit brings up what many call an old issue: the unauthorized practice of law by machines. Law professors bring up the Janson v. LegalZoom case, in which a Missouri federal court found that a class of consumers could move forward with allegations that fill-in-the-blank software had engaged in the unauthorized practice of law, because it offered a service: the automated preparation of documents, which were reviewed by nonlawyer employees who copy-edited them. The judge wrote that the fact the document was "prepared using a computer program rather than a pen and paper does not change the essence of the transaction".
As one law professor noted:
"Now you have these very powerful, large language model AI things out there that can provide legal advice or draft pleadings, or write nasty-grams to insurance companies. Clearly, if a human did those things, it would be the practice of law. So what's the status of a machine doing the same thing?"
Access to justice advocates have long worried that unauthorized practice of law regulations are too broad, limiting tools for self-represented litigants. Stanford law professor David Engstrom said:
"The requirement that only lawyers can practice law is there to ensure that legal services provided to people are high quality. Why? Because bad legal services have all kinds of very clear consumer protection implications. But the problem with any licensure system is access. It can also become a mechanism for protectionism and a mechanism for propping up the earnings of that profession."
But advocates are split on the best uses of artificial intelligence. Many point to a 2023 law journal article advocating for a new, less vague definition of the unauthorized practice of law, one that would allow for technologies and services that improve access to legal help.
But many others say we're in a big, transformative time in our society due to AI, and the legal system is no exception to that, so there needs to be guardrails put in place.
Many of these questions now hinge on whether a chatbot should be defined as a tool, a legal adviser or a third party. Courts have grappled with this question as they've struggled with issues of privilege in recent months as we noted in yesterday's post.
It's unclear to what level Nippon has successfully subpoenaed or acquired Dela Torre's ChatGPT history, though its complaint's allegation that she asked the machine if her attorney was gaslighting her suggests Nippon had some inside knowledge. Dela Torre did lament in a September court filing that Nippon was "demanding access to my private login credentials" in pursuit of its $10 million claim against OpenAI.
She added that the claim "targets a tool specifically designed to help individuals like me: pro se litigants trying to navigate the legal system without the benefit of legal counsel."
While the right to counsel for criminal defendants was enshrined by the Sixth Amendment and the U.S. Supreme Court's 1963 Gideon v. Wainwright case, no equivalent right exists in civil litigation. The Legal Services Corporation — a nonprofit offering civil legal aid to low-income court users — reported in 2022 that it receives 1.9 million requests for help each year, half of which are turned down due to limited resources.
Filings of pro se civil lawsuits seem to be on the rise, though according to an analysis by the National Center for State Courts, as of 2023, they had not yet returned to pre-pandemic levels. The NCSC study, which analyzed new cases in 28 states, found the upswing was mostly driven by contract disputes, which saw a 21% year-over-year increase in 2022 and a 15% increase in 2023.
Another analysis, by Fisher Phillips of new lawsuits in federal court and some state jurisdictions, found a 49% increase in pro se employment cases between 2024 and 2025.
Some scholars have argued AI could have a democratizing effect and improve access to justice, but others urge caution. The feeling? Access to bad legal help isn't really access to justice.
But you are still stuck with a legal system that needs to come to terms with the fact that in many cases, pro se defendants find themselves up against institutional plaintiffs — like banks, corporate landlords or the government — and some people are simply priced out of the market for civil legal services. If their alternative is a tool that is less than perfect and yet might actually allow them to navigate their case in court, then many think that's something that policymakers should be taking account of.
But the question of how to advise laypeople on using AI for legal services could be rendered moot, if the Nippon case becomes the first of many dominoes to fall.
| 
