Evaluating and Measuring Impact

Evaluation may not be your first priority as a creative placemaker, but it’s a critical part of any project—it can even be fun!

As evaluations work to acknowledge success, they provide encouragement to participants and partners, lay out a roadmap for improving programs, and establish accountability. Simply put, it’s imagining what difference you intend to make, and then collecting the stories, images, opinions, and other evidence that show how things are going.

Use the resources in this section to help you incorporate holistic documentation and evaluation from the beginning, so when it’s time to share your successes with stakeholders you have the data and materials you need.

WATCH: 

Crafting an Evaluation Plan & Measuring Impact
This video focuses on project evaluation. Evaluation can feel academic or daunting, but it can be one of the fun parts of the project and process. Simply put, it’s imagining what difference you intend to make, collecting stories, images, opinions, and other evidence that shows that you’re making a difference (or not), documenting that evidence with stories, images, and numbers, and reflecting on what it all means. Urban designer and planner, Lynn Osgood, will share on what makes a solid evaluation plan!

Evaluations work to:

• Acknowledge success respects and encourages participants and partners
• Improve programs
• Provide accountability for funders and partners
• Give cause for celebration
• Evaluation need not be mysterious. Simply put, it’s imagining what difference you intend to make, collecting stories, images, opinions, and other evidence that shows that you’re making a difference (or not), documenting that evidence with stories, images, and numbers, and reflecting on what it all means.

Evaluations help you answer:

✔ Are we doing OK?
✔ What difference did we make?
✔ What did we achieve?
✔ What worked well?
✔ What could we do better next time?
✔ What’s next?

Defining evaluation terms

As we outline the basic steps of organizing evaluation efforts, we begin with defining the essential terms.

• Activity: The individual project or program components that people participate in (e.g. a community dinner)
• Outputs: Generally, answer the question “What difference will the project make?”. These changes are framed in terms of the specific actor (i.e. participants, residents, families, organizations, systems, etc.) and the specific area of change (i.e. awareness, actions, capacity, attitudes, etc.)
• Outcomes: These are the quantifiable results achieved by a project—they identify specific activities done by a specific group of program participants. An output could also help to quantify outputs produced, or programming offered. Because outcomes are a basic measure of quantity, they are necessary for understanding the success of a program.  Though outcomes alone are not sufficient.
• Short-term outcomes: the immediate effects of the project
• Intermediate outcomes: initial effects that linger over time (e.g. “In the first six months of our project we saw....”)
• Long-term outcomes: the longer term goals of the project may be difficult to identify early on, such as decreased neighborhood violence, improved economic viability, or increased social cohesion, but that can be referenced when framing earlier outcomes
• Indicators: These are qualitative (numbers) or quantitative (description) measures of a specific outcome. Indicators are capable of being measured, through observation, conversation, reflections, etc., but also seek to answer the question, “How will we know when we’ve achieved our outcome?”. Often an outcome can have several different indicators associated with it.

Through the NEA creative placemaking program, Our Town, grants supporting art and design are awarded to transform political, economic, and social outcomes of communities. A logic model and theory of change are two tools utilized by program managers to describe and explain how a program is expected to proceed. The Our Town theory of change and logic model were developed through analysis of preexisting data on grantee applications and reports, interviews of national experts in the creative placemaking field, and study of place-based programs and resources. Our Town operates under a theory focused on systems change, as part of a larger nationwide effort to promote arts and culture in community development. The logic model for Our Town describes the program operation on the individual grant level. This includes everything from the context for the project, activities introduced during the project, and the final outcome of the specific project.

For more information please refer to the Our Town Theory of Change.

The evaluation process

Once you have the right person or people in place to help with the evaluation process, you can begin walking through the different steps:

• Identify what you know already: Chances are, you already know quite a bit about the project, its context, and even the initial impacts it’s making. Even the project grant application itself can be a good source for forming the initial narrative. What you already know will become part of the larger story that the evaluation process will help to tell.
• Determine what you need to know: At this stage, the project team should discuss the kinds of outcomes you imagine seeing and the specific audiences the final report / story will need to speak to. Different audiences will need information in different formats. Policy makers might appreciate more quantitative results, funders might like a mix of qualitative and quantitative information, and local community advocates may feel the stories of impact are the most powerful. The audience you are trying to reach should inform the kinds of impacts you look for and how you present them. Creative placemaking evaluation isn’t a controlled scientific investigation. It’s about finding out specific information that will be helpful to specific people.
• Gather the information you need: Ideally this step happens throughout the life of the project. Many of the project activities you’re already doing for other aspects of your project, such as interviews as part of community engagement, can equally contribute to project evaluation efforts. Information gathering techniques include:
  • Story gathering
  • Interviews
  • Focus groups or group conversations
  • Polls and surveys
  • Social media analytics
  • Mapping including asset mapping and network mapping
• Organize, analyze, and synthesize findings: At the end of the project — or even during the project — you’ll want to step back and look at the information you have. What story is beginning to emerge? Is there a pattern to the types of impacts that you’re seeing? Are certain types of evidence more helpful than others? This can be one of the most exciting parts of the evaluation process — take the time to work through ideas as a larger project team.
• Tell your story: With insights, timelines, and data in hand, you now have a larger story to tell. Remember stakeholders can be impacted by the story of your project in different ways:
  • Team Members — can reflect on lessons learned, and find ideas for next steps, or the next project.
  • Community Members — can incorporate the project narrative into the larger neighborhood narrative and find insights for future challenges they would like to address;
  • Q&A with Elected Officials — can bring public recognition and support for future efforts.
  • Creative Placemaking Practitioners — can learn from the project and become inspired for ideas they might want to bring to their own local communities.

Evaluation in plain English

What changes will your program make in the world?

• Outcomes
• What evidence would you accept?
• Indicators
• Did your program make the difference you intended?
• Outcome-based program evaluation

Community Engagement is a Core Asset:

• Designing
• Implementing
• Evaluating and
• Telling your story about Creative Placemaking

The Evaluator as an Artist:

• Creativity in research design
• Creativity in interpretation of findings
• Creativity in presentation of findings
• Creativity telling the story of result

Measuring evaluation impact

Things to consider:

• What is the change the community wants to see?
• How will we know when/whether the change happens?
• What do we need to measure to know?
• How can we measure?
• Who will measure?

Best practice: involving community and the arts in evaluation
While the project evaluation may be about impacts of this art-infused work on your community, it is also possible to bring the arts into the evaluation process. Ideas include:

• Holding a community dinner at the end of the project and giving attendees guiding questions to reflect on together.
• Facilitating a creative process for community members to express the changes they felt such as with storytelling, theater/improv exercises or visually.
• Train student youth to interview community members as a part of a short film project.
• Create a walk through the neighborhood that captures changes that people see (or that will happen in the future) and work with a graphic artist to record those in a “neighborhood map.”

Best practice: simultaneous evaluation
At the beginning of any creative placemaking effort, the project team should address how evaluation will be incorporated alongside the artistic process. To avoid duplicating efforts, or asking too much of the community members, it’s essential to harness everyone’s efforts for multiple purposes. Here are some ideas on how to integrate evaluation throughout the artistic process:

• Record interviews done as part of the community-focused artistic process in order to have access to them later as part of your larger evaluation /reflection process.
• When they emerge in the field, have the artist(s) text or email insights to the person in charge of capturing information. You’ll be able to search for these emails later to have a record of what your thoughts were in the moment, and you’ll help support a longer-term conversation about what is happening within the project.
• Hire and train local youth and community members to interview folks that participated in the project, asking what peoples’ impressions were and how it impacted them. You’ll learn about the impacts they discovered and hear their own insights into the project. Through their eyes, your team can also learn things they may have been unaware of before or about the scale of personal impact the project had on them.

Best practice: how to share your data

• Make sense of numbers quantitative data analysis
• Make sense of words & images
• Qualitative data analysis
• Reflect on what you’ve learned
• Summarize into key lessons learned and recommendations for improvement next time
• Create a word cloud using field interviews (Figure 1)
• Develop a colorful pie chart with survey responses (Figure 2)

EVALUATION WORKSHOP PRE-TEST & POST-TEST
→ Download the Worksheet

DESIGN YOUR EVALUATION PLAN
→ Download the Worksheet

Additional Resources

Curated for you by the LISC team, the resources below provide models of best practice, insights from our Resource Team, and other extras we hope will be useful to you as you navigate your creative placemaking projects.  

LEARN:

Resources on Program Evaluation and Performance Measurement, NEA Office of Research and Analysis
Compiled by the NEA Office of Research and Analysis, this site includes introductory resources on program evaluation for arts organizations, sample evaluation efforts from past Creative Placemaking projects.

Continuum of Impact Guide, Animating Democracy
This digital guide from Animating Democracy defines the social and civic outcomes that creative placemakers aspire to in their work. Use this guide to help you craft an evaluation plan that’s clear, realistic, and sensitive to the intangible outcomes that often and importantly define creative placemaking work.

WATCH: 

Crafting an Evaluation Plan & Measuring Impact
How does a creative placemaking evaluation plan work and what does it measure? This webinar explores best practices in creative placemaking evaluation techniques and help participants create or refine their own evaluation strategies for their projects.