"Eval" - Another 4-letter Word

By Karin Hostetter

For many museum staff, "eval," short for evaluation, is a four-letter word. You either love it or hate it. Grants require data obtained from evaluation and administrators want proof that programs "work," but evaluation takes time and is hard work. (Note: In this article, I am using the word "evaluation" broadly to mean any intentional attempt to gather information for the purpose of bettering a program, exhibit, or written interpretation. This article is focused more on formative than summative evaluation. Most everything can be done inexpensively in-house, though all of the basic principles apply for summative, contracted, and formal evaluations as well.) Most often, the cause of evaluation distress is that the questions used are not the ones which give the desired information. A few simple pointers can relieve the distress and provide information that helps everyone arrive at answers and improvements.

Know What You Want to Know

What do you want to know? What is the "better world" you want that points to evaluation as a possible solution? The more specific the need, the better the evaluation process will be and the more meaningful the results. Common needs include demographic information, whether visitors learned content, overall visitor feelings and thoughts and understandings, whether visitors enjoyed their visit, whether visitors changed behaviors, and data for grant reports. The "cause" should grow from the mission statement of the organization as well as the goals and objectives of the project. (Those basics of mission, goals, and objectives reappear to keep us on the right track.)

Be as specific as possible. If you want to know whether people "liked" a program or exhibit, know what "like" means to you and make that the cause of the evaluation. Maybe you want to know whether visitors had quality family time during a visit or felt energized after an experience. Specifically described causes will guide the choice of evaluation questions much better than causes that are too general.  

Use the Right Collection Method

Many different methods of gathering evaluative information exist. The challenge is to choose the right one for each situation. Most people think first of the paper or online survey. This is a good approach in many situations, but it is very impersonal and does not easily allow the ability to probe more deeply into an answer. If you want to get some brainstormed ideas or really probe a topic, you might be better served with a focus group.   If you want a large sampling of responses, a few questions on a quick survey will get you what you want better than the focus group or interviews. As you choose the right approach, you need to consider the time available to administer the survey, to tabulate results, and to analyze data. Remember to consider the budget needed to carry it all out.  

Ask good questions

This is where many evaluation attempts fall short. Writing questions that fit the chosen approach and really probe into what you want to know takes practice. Some aspects to consider include the following:

1. Ask questions that really probe into what you want to know. Present the same idea in a couple of different ways in different questions. Approach the concept from different angles so that people are urged to think about what you want to know in different contexts.
2. Avoid yes/no questions. They do not tell you much. They might be good if you need to gather some quick satisfaction data for a grant report or to share with a funder, but they do not help much with program or exhibit improvement.
3. Open-ended questions give users the opportunity to give their thoughts without being led to a conclusion, but using too many of them causes respondents to quit answering as it takes too much time to think and to fill out. If your survey is all scaled choices or rank the statements, then you are missing the opportunity to get opinions from guests that you did not think of and write into the process. At least allow an occasional opportunity for "other" or "explain" or "what else would you like to tell us." Often, these are the questions that give you the most insight.
4. Generally, a variety of questions is a good idea, grouping questions of the same type together.
5. On scaled questions, give an even number of choices. When an odd number of choices is available and participants do not know the answer or are ambivalent, they always choose the middle, neutral, answer. If you have an even number of choices, you force them to lean one way or another in their answer which gives you more information with which to work.

Above all, be clear, specific, and short. Make the entire process easy for you and for everyone participating. If you are just developing your evaluation skills, one suggestion is to ask a friend to answer the questions using the shortest possible answer since that is how most people answer questions. If you do not get the information you wanted, rewrite the question.

Read Results with a Critical Eye

Analyze the information gathered and see what it tells you. Be open to criticism so improvement can follow. Knowing which answers to focus on and which to gloss over takes practice and a hard skin sometimes. Look for patterns and repetitive answers. These are the ones that warrant attention. Outlying responses, both good and bad, will always occur, but avoid making major decisions based on those responses. They might be indications that more information should be gathered around that specific thought or they might bring up situations you cannot control.

Also use responses to indicate aspects that should not be changed. If an open-ended question on "what did you like best about today's program" leads a majority of the participants to write in "the game on how pioneers packed their wagons," then that is a component of that program which should be kept and maybe even given more time. When you do have questions that give you all "good" or "yes" responses, they do not tell you much about improvements, but they can be turned into statistics for reports such as "95% of respondents like the program and would refer a friend."

Be Aware of these Common Mistakes

No clear focus - Many surveys seem to ask a lot of questions covering all kinds of topics. This could be intentional, but more often it is the result of not defining clearly what it is you want to know. As the person taking the survey, you never get to settle your mind and think about one idea long enough to give good feedback. Several short surveys, each with a few questions on one or two topics, is generally better than one longer one that jumps to all kinds of topics. Know exactly what you want to get from the evaluation.

Questions too broad and not meaningful - This often follows from having no clear focus. The resulting questions are often so open-ended that people do not know where to start with their responses so they start giving answers of a few words and then just skip the questions. Alternatively, the response choices are so narrow, that the respondent does not get to clarify the choice (i.e. yes/no or agree/disagree).

Survey is too long or too short - This also follows from having no clear and defined focus. A good general rule is to try to keep to one page or ten questions for most general purposes. Two pages can be okay occasionally, but many people will get antsy when they see the second page looming. Think about matching the length of the survey or interview to the situation. Two pages is not good for a mom leaving the site with two kids in tow, but might be fine received as an email at home to do after the kids are in bed.

Data gathered is not studied for lessons learned and/or not applied - Then why do evaluation in the first place? Sheets of paper in a pile or a compilation of data in a computer file will not help programs and exhibits improve. Put the information you gather to work and use it to guide decision making for the future.

Developing good evaluation instruments takes practice. Remember to define clearly what you want to know; determine the best method to gather the information; develop strong questions which will result in usable information; analyze the results and use them to guide decision making. Our visitors deserve the best we can give them based on sound evaluation practices.  

Karin Hostetter has over thirty years experience with museum education. With a career that includes natural history museums, cultural history museums (including first person interpretation), nature centers, and zoos, Ms. Hostetter is experienced in interpretive writing, program and curriculum development, and staff and volunteer training. Ms. Hostetter is owner of Interpret This, a consulting company specializing in interpretive writing, program and curriculum development, and volunteer program management. When she is not consulting with other museums, she likes to volunteer and contract teach at them with a special love for preschool and family programs. Karin teaches a number of courses for museumclass.org including MS237 Formative Evaluations for Exhibits and Public Programs.

Mission and Exhibitions

By Lin Nelson-Mayson

Start with the mission

All museum activities should begin with the museum's mission statement, including exhibitions. When a museum evaluates the relative merit of exhibition and program ideas, the mission is the guiding point for decision-making and resource allocation. Exhibitions are the museum's main communication vehicle and certainly one that utilizes a large amount of a museum's resources. Starting with the mission enables the museum staff to ground ideas in a solid relationship with the institution and develop clarity of communication about the exhibition message.

Although objects are the central focus of a museum collection, an idea or message is generally the starting point for an exhibition. In fact, many exhibition proposals or scripts begin with the central theme or, as Beverly Sorrel terms it, the Big Idea. The Big Idea is the essence of the exhibition, the take-away message, as it were. An object, such as a teapot or chair, can be the source of this message, or it can be a concept, such as innovation or racism. In evaluating ideas for potential exhibitions, articulating the Big Idea can be a valuable exercise in clarifying the idea. An inexpensive evaluation technique is to ask a visitor to state his or her perception of the exhibition's Big Idea. It may not be the one intended by the exhibition team! Like a museum's mission statement, the clear central message, or Big Idea, helps those working on the exhibition make appropriate choices that convey the message and helps to focus the exhibition components to achieve it.

The Big Idea is necessary to provide a starting point for all the other elements of an exhibition, but the objects and exhibitry presented are the true heart of the exhibition. Objects can be a museum's greatest strength and the primary reason visitors seek out the experience. Object lists can be developed after the script is written, but a preliminary list is usually in progress, if not completed, to inform the development of the interpretive materials. The final selection depends on the final script, the object's availability and condition, and the limitations of the gallery space.

The physicality of the space in which the exhibition is presented is an element that affects the viewer, too. If the space is appropriate, it is rarely noticed. Spaces that are too small, too dark, too hot, or too distracting in other ways detract from the exhibition's message and can hamper communication and enjoyment of the intended exhibition experience.

Enhance and/or support collections

Exhibitions are often defined by the length of time they are open for viewing. Long-term exhibitions are generally presented to the public for a year or more. Temporary exhibitions may be displayed for as little as a few weeks. A museum may utilize both at once or concentrate only on one.

Most museums have collections. Long-term exhibitions are composed almost entirely of objects from the collections and form the heart of a museum's exhibition program. These exhibitions provide the central stories and experiences for museum visitors, the experiences that form a critical part of the museum's identity and the foundation for many related decisions regarding program and support areas. (For example, think of the names of museum restaurants you've visited - "The Flight Zone" in an aviation museum or an art museum's "Still Life Table.")

These long-term collection exhibitions are often the most significant investment in exhibitions undertaken by the museum. They can present the primary message of the museum to the community and provide a resource for frequent school visits to achieve curricula. In a building project - new or renovation, they inform the most fundamental decisions regarding architecture, image and preservation needs.

While a museum's collections support its mission and can form the backbone of the visitors' experience, temporary exhibitions can support these collections by offering a larger range of objects and subjects than may be contained in the collections. Temporary exhibitions can reflect new achievements or developments in areas of interest to the museum, they may include objects that are too large or too expensive for the collections, and they may explore subjects that are related, but not central to the museum's mission. While long-term exhibitions form the core message and experience presented by the museum, temporary exhibitions support this message with a fresh look and can attract new visitors to the museum based on an expanded subject or time-sensitive viewing opportunity.

Support strategic plan

Fundamentally, all exhibitions are developed as part of a comprehensive planning process that includes educational goals, financial planning, community outreach and partnership, and collections stewardship.

This planning process provides the framework for decision-making about exhibitions and their importance to the museum. Who are the audiences we are trying to reach? What resources do we have to achieve our goals? A regularly reviewed strategic plan can assist museum staff in forming a philosophical basis for exhibitions, in selecting exhibitions for development, and in determining what exhibitions to change.

Excerpt from MS106: Exhibit Fundamentals: Ideas to Installation.  

Lin Nelson-Mayson, with over 25 years of museum experience at small and large institutions, is director of the University of Minnesota's Goldstein Museum of Design. Prior to that, she was the director of ExhibitsUSA, a nonprofit exhibition touring organization that annually tours over 30 art and humanities exhibitions across the country. For five years, she was a coordinator or judge for the American Association of Museums' Excellence in Exhibitions Competition. She currently serves on the exhibition committee for the National Sculpture Society. Ms. Nelson-Mayson has extensive experience with the planning, preparation, research and installation of exhibitions. Ms Nelson-Mayson's experience includes teaching museum studies and museology courses. Her particular interest is the needs of small museums.  She teaches MS 106: Exhibit Fundamentals: Ideas to Installation and two other courses for museumclasses.org.