|
Before using structured outputs, answers would be formatted in different ways despite attempts to prevent that in the prompt. Those formatting irregularities would break the processing pipeline.
We queried the LLM in bulk using scripts written in Python. SQLite databases were another very handy tool to keep track of LLM responses and store metadata about each LLM query. For the actual analysis of the responses, we used R.
People might assume AI integration in journalism means less people are involved in the work — but we counted 19 reporters, editors, and artists involved in this story in some capacity. That’s an impressive number! How did you divide up the work, and why were that many individuals necessary to complete this project?
Harki: It took a team of journalists to do the story because we used AI. The story had six bylines and every one was very much deserved. Alex Ebert and Kimberly Robinson, who cover federal courts and the Supreme Court, served as our in-house experts on the law. Diana Dombrowski, an investigative reporter, was lead writer and did much of the work forming the questions we asked the LLM to create the dataset. About 20 people from around the newsroom helped us validate the data, making sure the LLMs answers were correct. In the grand tradition of newsrooms everywhere, we fed the staff pizza in thanks for the work.
Umar Farooq, Sophie Will, and Jon Meltzer were involved in both LLM and data work and Meltzer created the graphics and presentation with our art director David Evans. We had two outside journalists advising us as well. Derek Willis, who has worked at ProPublica and the New York Times, among other places, and is now a professor at the University of Maryland teaching data and computational journalism, was our advisor on all things LLM. Seamus Hughes is a well-known expert on the federal court system and was a big help anytime we had to puzzle out something dealing with federal court filings.
So that’s all the work on just creating the dataset using the LLM. And then, of course, all the normal reporting things had to happen once we had the results. There were also several engineering teams outside of the newsroom that work on Bloomberg Law’s dockets database that were a big help. One team pulled the data from PACER and made sure we had all the cases, another helped us get the initial complaints scanned so we could use an LLM to read them. The story had us working across teams. Bloomberg is a big place, and it’s amazing we have access to all of these resources and talented people.
Coming up next week: Harki and Wallender share ethical considerations when using LLMs in reporting as well as top takeaways from this project.
| 
