Greetings!
In just the last month since we wrote to you, it seems like artificial intelligence (AI) has taken over our conversations! It comes up in our personal and professional lives. I'm sure like us, you are trying to understand how to use AI tools for evaluation, and how to ensure equity stays front and center despite these shiny new toys that threaten to distract us and add complexity to our cause.
This month, we share some resources we are reading (and revisiting) to guide us through evaluation's growing edge with AI.
|
|
Opportunities for AI in Evaluation
Are we at a fork in the road?
|
|
To date, evaluations of and with artificial intelligence (AI) have largely been underdeveloped and under-explored within the field of evaluation. This symposium, hosted by New Directions for Evaluation Co-Editors-in-Chief Drs. Bianca Montrosse-Moorhead and Sarah Mason, explored the implications of and opportunities for artificial intelligence in evaluation, pondering the questions:
- What role should artificial intelligence play in evaluation? What are the implications for the field? What challenges does emerging AI pose?
- Which parts of evaluation are inherently human? Which parts, if any, can and should be passed on to an AI chatbot? How much will evaluators and evaluation users trust the products of AI-generated or AI-assisted work?
- How do we train emerging evaluators to work in a world in which AI is prevalent?
Our team watched the whole recorded symposium, and we recommend it as a good basic primer and for some practical ideas and tips for including AI in your evaluation practice.
|
|
Automating Inequality
Algorithms are not objective
|
|
As our team reads about and experiments with AI, we are reminded also of its limitations. In her (older by still great) talk at the University of Michigan, Virginia Eubanks reminds us that automated systems - like the algorithms AI are built on - are subject to same biases and systemic injustices of their human creators. Sharing insights from her book Automating Inequality: How High-Tech Tools Profile, Police and Punish the Poor, Eubanks discusses inherent challenges with the automated systems employed in our society such as 'feedback loops of injustice,' and offers some potential solutions.
|
|
Using AI for Content Analysis (and more)
Examples for using AI in practice
|
|
Our team has been experimenting with AI tools such as ChatGPT, and many of our colleagues are as well! This paper, written by AI and international M&E practitioners such as the World Bank, "presents the methodology and results used to pilot and test the applicability, usefulness, and added value of using artificial intelligence for advanced theory-based content analysis." Basically, they use AI to evaluate a "conceptual framework" (also known as theory of change). We admit, the language is a bit technical, but they help break it down with some visuals comparing a human and AI-led approach. The detail about their methodology adds lots of technical insight into how AI could be used.
If looking at specific examples for how evaluators have used or experimented with AI are helpful, we also recommend following Rick Davies's blog.
|
|
AI Tools for Evaluation
We haven't experimented with all of these tools yet, but here are some on our radar for further exploration. Let us know on LinkedIn how they work for your practice!
|
|
Ailyze
AI tool for qualitative analysis. Can summarize, answer questions, get themes, and more.
|
|
Consensus
AI tool for research. Extracts and distills themes from scientific research.
|
|
Elicit
AI tool for lit review. Elicit can find relevant papers, summarize takeaways, and extract key information.
|
|
We are becoming more equitable, both within our organization and through the way we work with our partners. In case you missed it, read the transformation letter from our Board and Consultant Team and our new Vision and Values. Follow our transformation in this blog series. |
|
|
|
|
|
From the Innovation Network Team |
|
|
Follow us for more free resources! |
If you aren't already, follow us on Twitter @InnoNet_Eval or LinkedIn @Innovation-Network. We'll keep you up to date on the latest trends, articles and comical musings coming out of the field of evaluation, and our annual conference. |
|
|
|
|
|
|
|
