|
Wallender: We approached this project with a healthy dose of caution. We know AI can hallucinate or misinterpret questions, so we had to be completely confident in the LLM’s output.
Part of the reason the story took so long was the amount of time we spent repeatedly testing and re-writing the prompts to ensure accuracy. It was a good exercise in being really deliberate with the language we use to define certain attributes. Checking the LLM for accuracy forced us to see our story in a new light by making us break down each definition we used and contemplating how we wanted to categorize cases that didn’t exactly match initial definitions we established.
We set predefined thresholds for accuracy so that we could be confident in the figures cited in the story. We then had a small army of reporters offer up their time to read through a random sample of cases and answer the same questions we asked the LLM, so we could assess accuracy. At least two reporters provided responses for each case. When people disagreed on answers, we scrutinized the responses and rationale to see if there was a mistake or we needed to discuss as a group the correct answer.
Thankfully, our newsroom and company as a whole are no stranger to AI. There are a number of people at Bloomberg Industry Group and Bloomberg LP who have been working with AI for years who offered us advice and best practices. Our company also has an extensive generative AI policy and requires that any use of the technology, including for this story, is approved by a committee of reviewers.
Read the first part of this interview for key takeaways from their investigation.
|