Interactive Evaluation Practice
Presenters:
Jean A. King
University of Minnesota
Organizational Leadership,
Policy, and Development
kingx004@umn.edu
(612) 626-1614
Laurie Stevahn
Seattle University
Educational Leadership
Doctoral Program
stevahnl@seattleu.edu
(206) 296-2559
Overall Session Goals
1. Examine frameworks that ground
Interactive Evaluation Practice
(IEP).
2. Experience strategies for
facilitating IEP and rationales for
using them in evaluation studies.
3. Consider implications and
applications of the strategies
for your own evaluation practice.
Shape of This Afternoon
 Introductions
—you, us, we
 Frameworks
—for Interactive Evaluation Practice
 Evaluator’s Dozen of
Cooperative Strategies
—responding to set content
—generating information
—organizing information
Jean’s background . . .
• Roles ► teacher, teacher educator,
director of a collaborative research center,
professor of evaluation studies
• Education ► English, curriculum and
instruction, research design
• Specialties ► participatory evaluation,
evaluation capacity building, evaluator
competencies
• Passions ► family, friends, program evaluation,
kittens/cats (anything that “meows”)
Laurie’s background . . .
• Roles ► teacher, consultant,
researcher, evaluator, professor
• Education ► political science (BA),
curriculum and instruction (MEd),
educational psychology (PhD)
• Specialties ► cooperative strategies,
constructive conflict, evaluator competencies,
organizational change, inquiry methods
• Passions ► faith, family, friends, music, art,
creativity, culture, collaboration, travel
Your background . . .
Strategy #1: Voicing Variables
How long have you been involved with evaluation?
 Less than a year
 1-5 years
 6-10 years
 More than a decade
Strategy #1:Voicing Variables
In what fields/contexts do you work?
Healthcare
Education
Social service
Government
Nonprofit
International
Other . . .
Strategy #1: Voicing Variables
Your role as an evaluator . . .
Internal
External
Both
Strategy #12: Fist to Five
What is your experience with . . .
• Qualitative studies
• Quantitative studies
• Mixed-methods studies
Strategy #12: Fist to Five
What is your experience in conducting . . .
• Single-program/organization evaluations
• Large-scale and/or multiple-site
evaluations
• Community development and/or
grass-roots evaluations
Strategy #5: Round-Robin Check-In
 Form groups of three
 Introduce yourself
 Your name
 Why you want to learn
about interactive
strategies
Grounding Frameworks for IEP
1. Basic Inquiry Tasks (BIT)
2. Interactive Participation Quotient (IPQ)
3. Evaluation Capacity Building (ECB)
FIRST FRAMEWORK — Exhibit 2.1 (p. 23)
Basic Inquiry Tasks (BIT)
1.
2.
3.
4.
5.
6.
7.
Framing questions (focusing the study)
Determining an appropriate design
Identifying samples (sources of information)
Collecting data
Analyzing data and presenting results
Interpreting results
“Reporting”
What are possible evaluator-client
roles and relationships in carrying out BIT?
• A relationship exists between the
evaluator and client—i.e., program leaders,
staff, funders, community members, other
evaluation stakeholders
• Involvement in evaluation decision making
and implementation may shift between the
evaluator and client/stakeholders during the
study
SECOND FRAMEWORK — Exhibit 2.3 (p. 27)
Interpersonal Participation Quotient (IPQ)
HIGH
Involvement in
decision making
and implementation
Evaluator
Program
leaders,
staff,
community
members
LOW
Evaluatordirected
Collaborative
ZONES
Participantdirected
Diverse evaluator roles . . .
• Technical expert on
evaluation research design,
measurement, coding, data analysis
• Facilitator of group interaction
• Coach of others doing their own
evaluations
• What else?
THIRD FRAMEWORK — Exhibit 2.6 (p. 35)
Evaluation Capacity Building (ECB)
Formative / Summative
evaluation study
Evaluation
specifically for
building capacity to
evaluate
Evaluation for
organization
development
◄-----------------------------------------------------------------------►
Use of single study
process / results
ECB = creating
capacity to conduct
evaluations
Capacity to sustain
change / continuous
improvement
IEP highlights the importance of . . .
•
•
•
Involving people effectively—
constructive interaction with others is
essential to the success of all evaluations
Learning through evaluation—
interaction and participation should
promote learning
Building capacity to think evaluatively—
through meaningful engagement in
evaluation planning, acting, reflecting
An evaluator’s dozen of interactive strategies . . .
#1.
#2.
#3.
#4.
#5.
#6.
Voicing Variables
Voicing Viewpoints/Beliefs
Choosing Corners
Cooperative Interviews
Round-Robin Check-In
Making Metaphors
An evaluator’s dozen of interactive strategies . . .
#7. Data Dialogue
#8. Jigsaw
#9. Graffiti/Carousel
#10. Concept Formation / Cluster Maps
#11. Cooperative Rank Order
#12. Fist to Five
#13. Dot Votes / Bar Graphs
Different Types of Involvement
An evaluator’s dozen of interactive strategies for . . .
 Level I
Responding to set content
(#1-3, #8, #10-13)
 Level II
Generating information (#4-7, 9)
 Level III
Organizing or sharing
information (#8, #10-11, 13)
Strategy #2: Voicing Viewpoints/Beliefs
A. A good evaluator shares control of the study with clients
from start to finish.
1 Strongly 2 Agree 3 Disagree 4 Strongly
Agree
Disagree
____________________________________________________________________________________________________________________________
B. Program participants will bias a study if they are involved in
planning it.
1 Strongly 2 Agree 3 Disagree 4 Strongly
Agree
Disagree
____________________________________________________________________________________________________________________________________________________________
C. Program staff must be neutral during an evaluation study of
their own programs.
1 Strongly 2 Agree 3 Disagree 4 Strongly
Agree
Disagree
Strategy #13: Dot Votes / Bar Graphs
A. A good evaluator shares control of the study with clients
from start to finish.
Strategy #13: Dot Votes / Bar Graphs
B. Program participants will bias a study if they are involved in
planning it.
Strategy #3: Choosing Corners
C. Program staff must be neutral during an evaluation study of
their own programs.
Strategy #4: Cooperative
(three-step) interview
• Three roles create three steps:
– Interviewer
– Responder
– Recorder
• The interview process is
structured to build on
psychological principles of
cooperation
Interview Focus
TEAMWORK—it was terrific!
nInterviewer → seek input (respectfully)
• What made the experience so successful?
• When, where, who, what, why, how?
nResponder → tell your story
nRecorder → document (key words/phrases)
_______________
ROTATE ROLES
Similarities/Themes
________________________________
What does this mean for evaluators?!
Strategy #8: Jigsaw
1. Home groups divide information/readings
(Persons A, B, C, each get a different part)
2. Expert groups of two (all A’s, B’s, C’s)
read and prepare to teach
3. Back to home groups to present
4. Apply the entire body of information
Jigsaw Scenario . . .
• Large social service organization
• Provides numerous diverse programs for the community
• Assessing its current mission/vision/values in light of
changing social, economic, political, educational, and
technological concerns
• Evaluators hired to conduct ongoing meetings with
groups of stakeholders who are all in for the long haul
• “Pluses/Wishes” meeting reflections from . . .
– Program Directors (gold sheet)
– Program Providers (blue sheet)
– Program Recipients (green sheet)
Jigsaw process . . .
nHOME TEAM of three—each member gets a different
segment (gold, blue, or green sheet)
nEXPERT PAIR of two—find one other person in the room
who has your same segment (two golds together, etc.)
nRead the data and identify major themes
nReturn to HOME TEAM . . .
 Share major themes from each stakeholder group
 Compare/Contrast across all stakeholder groups
 Recommendations for future action?!
Strategy #9: Graffiti / Carousel
Strategy #10: Concept Formation
• Can be done on the wall,
informally, and is then
called graffiti
• Can be done on flipchart
paper passed among
groups and is then
called carousel
Strategy #9: Graffiti / Carousel
1. Face-to-face communication is most useful when . . .
2. Face-to-face communication is frustrating when . . .
3. Electronic communication works well when . . .
4. Electronic communication is problematic when . . .
Strategy #10: Concept Formation
_______________________________
1. Write one comment per sticky note,
as many as possible.
_______________________________
2. Organize “alike” items into
clusters/groups/themes.
_______________________________
3. Label each cluster/group/theme.
Strategy #11: Cooperative Rank Order
• An interactive social process for reaching
consensus on a rank order
• Sequence from most to least effective,
best to worst option, highest to lowest priority,
or some other continuum.
• Colored paper (strips)
can facilitate cross-group
comparison
Strategy #11: Cooperative Rank Order
Evaluators need support, too!
• Consider a list of practices to support the
professional development of evaluators
• Sequence the practices from
most to least helpful
• Be ready to explain/defend
your reasoning
1. Ongoing opportunities for formal
professional development near-by (e.g.,
attending trainings, local conferences)
2. Opportunities to attend national conferences
(at least one a year)
3. Regular informal gatherings with other
evaluators to discuss issues, problem solve
concerns, share “what works,” etc.
4. Reading professional literature on your own
5. Participating in an informal evaluation “book
club”
Strategy #7: Data Dialogue
• A process to use when
you cannot afford focus
groups
• It takes advantage of
some of the processes
of the three-step
interview
• Can be useful in
community settings
Strategy #7: Data Dialogue
• Purpose  To provide input on today’s workshop.
• Directions  Form groups of 3-4; decide who will write;
make sure all input is recorded (this is NOT about consensus;
it’s about getting everyone’s thoughts—seek and expect
diverse perspectives).
• Results  Your input will be kept confidential and will NOT be
traceable back to you.
• Participants Please indicate that you participated by
completing the information below, then detach this top sheet
and place in the envelope for confidentiality. Do NOT put
names on any other sheets. Thank you for participating!
Signatures: __________ __________ __________ __________
Strategy 7: Data Dialogue
TOPIC: Today’s Session
Pluses
Wishes
Strategy #8: Making metaphors
“A
is worth 1000 words.”
___________________________________________________________________________________________________________________________________
Tall
impressive
terrifying
magnificent
BIG
Find a picture to
finish this phrase . . .
“Interactive Evaluation Practice
will be successful when ________.”
An evaluator’s dozen of interactive strategies . . .
#1.
#2.
#3.
#4.
#5.
#6.
Voicing Variables
Voicing Viewpoints/Beliefs
Choosing Corners
Cooperative Interviews
Round-Robin Check-In
Making Metaphors
An evaluator’s dozen of interactive strategies . . .
#7. Data Dialogue
#8. Jigsaw
#9. Graffiti/Carousel
#10. Concept Formation / Cluster Maps
#11. Cooperative Rank Order
#12. Fist to Five
#13. Dot Votes / Bar Graphs
Review- Overall Session Goals
1. Examine frameworks that ground
Interactive Evaluation Practice
(IEP).
2. Experience strategies for
facilitating IEP and rationales for
using them in evaluation studies.
3. Consider implications and
applications of the strategies
for your own evaluation practice.
Thanks!
Jean & Laurie
Thanks!