|
25 April 2025 -- This is a follow-up to a story we posted from the Los Angeles Times this past February when the Times reported that hundreds of prospective California lawyers complained that their bar exams were plagued with technical problems and irregularities,
Yesterday, fresh outrage. The state’s legal licensing body admitted that some multiple-choice questions were developed with the aid of artificial intelligence. By non-lawyers.
Here are a few clips from the Los Angeles Times story and the Politico story, the latter having some background on the tech used.
The State Bar of California said in a news release that it will ask the California Supreme Court to adjust test scores for those who took its February bar exam.
But as the Times noted, it declined to acknowledge significant problems with its multiple-choice questions — even as it revealed that a subset of questions were recycled from a first-year law student exam, while others were developed with the assistance of AI by ACS Ventures, the State Bar’s independent psychometrician. Mary Basick, assistant dean of academic skills at UC Irvine Law School:
The debacle that was the February 2025 bar exam is worse than we imagined. I’m almost speechless. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable.
After completing the exam, Basick said, some test takers complained that some of the questions felt as if they were written by AI. Said Basick:
I defended the bar. "No way! They wouldn’t do that!" I told multiple test takers. Using AI-developed questions written by non-legally-trained psychometricians represented an obvious conflict of interest because these are the same psychometricians tasked with establishing that the questions are valid and reliable.
Basick's comments were echoed across California. Katie Moran, an associate professor at the University of San Francisco School of Law who specializes in bar exam preparation:
It’s a staggering admission. The State Bar has admitted they employed a company to have a non-lawyer use AI to draft questions that were given on the actual bar exam. They then paid that same company to assess and ultimately approve of the questions on the exam, including the questions the company authored.
The State Bar, which is an administrative arm of the California Supreme Court, said that the majority of multiple-choice questions were developed by Kaplan Exam Services, a company it contracted with last year as it sought to save money.
According to a recent presentation by the State Bar, 100 of the 171 scored multiple-choice questions were made by Kaplan and 48 were drawn from a first-year law students exam. A smaller subset of 23 scored questions were made by ACS Ventures, the State Bar’s psychometrician, and developed with artificial intelligence:
We have confidence in the validity of the multiple-choice questions to accurately and fairly assess the legal competence of test-takers.
A spokesperson for the California State Bar told the Times that all questions - including the 29 scored and unscored questions from the agency’s independent psychometrician that were developed with the assistance of AI - were reviewed by content validation panels and subject matter experts ahead of the exam for factors including legal accuracy, minimum competence and potential bias.
When measured for reliability, the State Bar told the Times, the combined scored multiple-choice questions from all sources - including AI - performed “above the psychometric target of 0.80".
The State Bar also dismissed the idea of a conflict of interest:
The process to validate questions and test for reliability is not a subjective one and the statistical parameters used by the psychometrician remain the same regardless of the source of the question.
Alex Chan, an attorney who serves as chair of the State Bar’s Committee of Bar Examiners, told the Times that only a small subset of questions used AI - and not necessarily to create the questions:
The professors are suggesting that we used AI to draft all of the multiple choice questions, as opposed to using AI to vet them. That is not my understanding.
Chan noted that the California Supreme Court had urged the State Bar in October to review “the availability of any new technologies, such as artificial intelligence, that might innovate and improve upon the reliability and cost-effectiveness of such testing". Chan said:
The court has given its guidance to consider the use of AI, and that’s exactly what we’re going to do.
That process, Chan explained, would be subject to the Court’s review and approval:
The Committee was never informed about the use of AI before the exam took place, so it could not have considered, much less endorsed, its use.
A spokesperson for California’s highest court said that justices found out only this week that the State Bar had utilized AI in developing exam questions. In a statement a court spokesperson said:
Until yesterday’s State Bar press release, the court was unaware that AI had been used to draft any of the multiple-choice questions.
Last year, as the State Bar faced a $22-million deficit in its general fund, it decided to cut costs by ditching the National Conference of Bar Examiners’ Multistate Bar Examination, a system used by most states, and move to a new hybrid model of in-person and remote testing. It cut an $8.25-million deal with test prep company Kaplan Exam Services to create test questions and hired Meazure Learning to administer the exam.
There were multiple problems with the State Bar’s rollout of the new exams. Some test takers reported they were kicked off the online testing platforms or experienced screens that lagged and displayed error messages. Others complained the multiple-choice test questions had typos, consisted of nonsense questions and left out important facts.
The botched exams prompted some students to file a federal lawsuit against Meazure Learning. Meanwhile, California Senate Judiciary Chair Thomas J. Umberg called for an audit of the State Bar and the California Supreme Court directed the agency to revert to traditional in-person administering of July bar exams.
But the State Bar is pressing forward with its new system of multiple-choice questions - even though some academic experts have repeatedly flagged problems with the quality of the February exam questions. Basick and Moran wrote in a public comment to the Committee of Bar Examiners:
Many have expressed concern about the speed with which the Kaplan questions were drafted and the resulting quality of those questions. The 50 released practice questions - which were heavily edited and re-released just weeks before the exam - still contain numerous errors. This has further eroded our confidence in the quality of the questions.
Historically, Moran said, exam questions written by the National Conference of Bar Examiners have taken years to develop.
Reusing some of the questions from the first-year law exam raised red flags, Basick said. An exam to figure out if a person had learned enough in their first year of law school is different from one that determines whether a test taker is minimally competent to practice law, she argued:
It’s a much different standard. It’s not just, "Hey, do you know this rule?" It is, "Do you know how to apply it in a situation where there’s ambiguity, and determine the correct course of action?"
Also, using AI and recycling questions from a first-year law exam represented a major change to bar exam preparation, Basick said. She argued such a change required a two-year notice under California’s Business and Professions Code.
But the State Bar told the Times that the sources of the questions had not triggered that two-year notice:
The fact there were multiple sources for the development of questions did not impact exam preparation.
Basick said she grew concerned in early March when, she said, the State Bar kicked her and other academic experts off their question-vetting panels. She said the State Bar argued that those law professors had worked with questions drafted by the National Conference of Bar Examiners in the last six months, which could raise issues of potential copyright infringement:
Ironically, what they did instead is have non-lawyers draft questions using artificial intelligence. The place the artificial intelligence would have gotten their information from has to be the NCBE questions, because there’s nothing else available. What else would artificial intelligence use?
Ever since the February exam debacle, the State Bar has underplayed the idea that there were substantial problems with the multiple-choice questions. Instead, it has focused on the problems with Meazure Learning:
We are scrutinizing the vendor’s performance in meeting their contractual obligations.
The State Bar provided a document that listed the problems test takers experienced and highlighted the relevant performance expectations laid out in the contract.
But critics across the California legal community accused the State Bar of shifting blame - and argued it has failed to acknowledge the seriousness of the problems with multiple-choice questions. Moran (among many others) called on the State Bar to release all 200 questions that were on the test for transparency and to allow future test takers a chance to get used to the different questions. They also called on the State Bar to return to the multi-state bar exam for the July exams.
The Committee of Bar Examiners will meet on May 5 to discuss non-scoring adjustments and remedies. But there is doubt that the State Bar would release all 200 questions or revert to the National Conference of Bar Examiners exams in July. The NCBE’s exam security would not allow any form of remote testing, and the State Bar’s recent surveys showed almost 50% of California bar applicants want to keep the remote option.
My two cents? The quiet adoption of AI to generate questions and expanding the scope of testing without adequate notice can certainly disadvantage candidates. If the State Bar believes in the integrity of these methods, it should openly communicate and justify them. Otherwise, it risks undermining the very fairness and preparedness it aims to test.
But this "quiet adoption" of AI is happening across so many industries I suspect it will come back to bite a lot of people on the butt.
| 
