Actionable Metrics for
Continuous Improvement:
Balanced Scorecard Survey Tools
Alexis Naiknimbalkar, CSU Chancellor’s Office
Angela Song, UC San Diego
Key topics for today
1.
2.
3.
4.
5.
6.
7.
8.
Introductions
CSU-UCSD Collaboration
Quick view of the Balanced Scorecard
The UC San Diego Surveys and Analytics Program
Overview of the Surveys
How the Survey results are used to drive improvements
The CSU – UC Collaboration Journey begins!
Next steps
The CSU-UC collaboration
• Background
• The collaboration
• UC San Diego’s surveys
• By using these standard questions, it enables benchmarking and
sharing of best practices between Universities and departments
First… A quick history of UCSD’s
Balanced Scorecard
What is the Balanced Scorecard?
History:
Purpose:
• Developed in 1993 by David P. Norton and
Robert Kaplan
• A framework for performance
measurement, communication, and
management tied to strategy
• More than half of major companies in the US,
Europe and Asia are using balanced scorecard
approaches
• Harvard Business Review rates the Balanced
Scorecard as one of the most influential
business ideas of the past 75 years.
Four Perspectives:
1) Financial/Stakeholder
2) Internal Processes
3) Customer
Survey!
4) Innovation and Growth
Survey!
• Provides a roadmap on where we
should focus energies, priorities, and
resources
• A holistic view of the organization
The Survey Program’s Evolution at UC San Diego
1997Staff
Climate
Survey
2003Surveys
go online
2004Surveys
expand to
all campus
2007First study
on diversity
and staff
satisfaction
1995 Student
Satisfaction
Survey
1993 –
1994 Customer
Satisfaction
Survey
16 Business
Affairs units
begin using the
balanced
scorecard
approach
UC San Diego
inducted into
Balanced Scorecard
Hall of Fame
2012 –
UC San
Diego’s 1st
strategic plan
2011 Scalable
advanced
analytics
and reports
Customer service,
department outreach,
and efficient internal
processes are an
expectation and norm
2014 Surveys
are fully
scalable
Surveys provide
actionable data for
continuous
improvement
initiatives and in
support of the
campus strategic
plan
Best practices and
benchmarking
opportunities
Key
Performance
Indicators
identified and
benchmarked
Mission and
vision aligned
with strategy
UC San Diego – VC Business Affairs
1994-2004
UC San Diego – All VC Areas
2005-2015
UC San Diego, UC Irvine (OIT), UC Riverside
(BAS), CSU Chancellor’s Office (BAF)
2016-beyond
LETS TALK ABOUT SURVEYS
FOR A MOMENT
Surveys are fun!
5 most bizarre survey finds
❶❷❸❹❺
From: oddee.com
5 most bizarre survey finds
#❶ 23% thought “MP3” was
a Star Wars robot
(– Vouchercloud.net)
5 most bizarre survey finds
#❷ 51% of surveyed
Americans think stormy
weather "affects" Cloud
Computing
(-Wakefield Research, 2012)
5 most bizarre survey finds
#❸ 1 in 4 Americans
thinks the Sun goes
around the Earth
(-NSF, 2012)
5 most bizarre survey finds
#❹ Average Americans
think they're smarter than
the Average American
(-YouGov, 2014)
5 most bizarre survey finds
#❺ Survey found that
most Americans (75%)
don’t trust Survey Results
(-Kantar data investment management poll, 2013)
Surveys allow a way to understand people’s
attitudes, feelings, and behaviors
Do I think this is a good place to work? Will I do my best to contribute or will I just coast? How will I talk about it to others
inside and outside the university? Do I feel valued? Do I feel I am making a difference? Am I feeling engaged and inspired or
bored and just clocking my time? Does X department even care if they are helping or getting the way of my work? ……
Four main reasons to survey customers
Uncover
Answers
1. Identify and fix
2. Assess the performance
Evoke
Discussion
3. Improve processes
Databased
Decisions
4. Understand needs for a better overall
experience
Compare
Results
Q. Why do (administrative support areas) survey in a university setting?
A. We should know if we are helping to support the mission of the university.
How would a leader know if he/she is meeting these needs?
ITS ALL
ABOUT
THE DATA
Getting good data from people:
-- Psychometrics -Not
METRICS
Psychometrics – getting good data from people
(because we cant read their minds!)
Psychometrics is a field of study concerned with the theory
and technique of psychological measurement.
 Construct and validate assessments instruments
(questionnaires, tests, personality tests)
 Statistical measurement theory
Psychometric research involves two major tasks:
1) Create instruments that are valid
2) Develop procedures to measure
Methodology – Design Customer Satisfaction Survey
Standard 8 rating questions for all services with up to 5 customized
questions per service area
1.
2.
3.
4.
5.
6.
7.
8.
Overall satisfaction
Understands my needs
Accessible
Responsive
Resolves issues
Knowledgeable/professional/courteous, etc.
Effective use of Blink (info sharing website)
Moving in positive direction
Stop, save, and finish later
Confidential responses
All staff and faculty invited
Message: Help us help you fulfill the mission of the university!
Methodology – Design Staff@Work Survey
Tested for internal reliability,
conducted Factor Analysis
53 questions measure 4 dimensions:
1.
2.
3.
4.
“Yes, it really is Anonymous” 
UC San Diego overall
Department effectiveness (diversity, mission)
Supervisor effectiveness
Employee effectiveness
Equity, diversity, and inclusion
questions for comparisons
Message:
•
•
Are you a satisfied UCSD employee?
Would you recommend others to work here?
Regressions analysis to predict what
drives satisfaction
Participation rate: 100% for some
areas, 56% overall
Annual Campus Survey Overview
Faculty and Staff Customer
Satisfaction
Student Customer
Satisfaction
• 21 years of data
• 20 years of data
• Departments voluntarily
opt-in to be rated
• Departments voluntarily
opt-in to be rated
• 80 service units/programs
currently evaluated
• 44 student service
units/programs currently
evaluated
• 3,773 Verbatim comments
and suggestions for
improvement
• 20,351 Verbatim
Comments
• One “burning question”
• One “burning question”
• Special Recognition for
Customer Service
• Net Promoter Score (NPS)
• Net Promoter Score (NPS)
• 36% responded with
varying participation per
rated unit
• 4884 undergraduate and
graduate students
participated
Staff@Work
(ie employee engagement)
• 18 years of data
• This is a campus wide
anonymous survey for staff
invited by the Chancellor
• Eight VC areas - 450 units
participate
• 824 Verbatim Comments
• “Who made a difference in
creating a positive work
environment?”
• Net Promoter Score (NPS)
• 56% responded campus
wide. Participation within
departments reach up to
100%
Congratulations! Here are your survey results. Good luck!
Analytics? Sure! Here you go.
Not cool.
ITS ALL ABOUT
making meaning of
THE DATA
Customer Satisfaction Survey Reports: Who, What, Where, When, Why
Trend analysis: “When” over time
Heat maps for easy identification of “Where” should we dig deeper
Correlational analysis
to identify drivers of satisfaction
and start the conversation of
“Why” are the scores varying
“What” are the basic descriptive statistics
Drill downs to know which departments
to focus your outreach or study best
practices: “Who” needs attention
Customer Satisfaction Survey heat map quickly identify
strengths and opportunities
Staff@Work: A picture can say a thousand words…
Impact analysis:
These are the items
where people are saying,
“I am not as happy about
these things and they are
also very important
drivers of my
satisfaction”
Descriptive statistics
Arrows indicate
positive or negative
movement and
statistical analysis
inform you of
significant trends
Interpreting the impact analysis report
“I am happy about these
things BUT they are not
things that are as important
to me in impacting my level of
satisfaction”
Keep it up! But no
need to spend too
much effort here.
“I am not as happy about
these things BUT they are not
things that are as important
to me in impacting my level of
satisfaction”
Keep an eye on these if
they move into
Primary Opps 
This quadrant shows the survey items that are rated
high (above the mean score of 3.73) and also found to
be highly related to satisfaction (above the mean
correlation of .42). “I am happy about these things
AND they are important to me in impacting my
satisfaction.”
Keep it up! Don’t
change anything
All is well.
“I am not as happy about
these things AND they are also
things that are important to
me in impacting my level of
satisfaction”
Focus on these things
to address!
Correlation means relationship. The closer to 1.0, the stronger the relationship between satisfaction and the survey item
Now what?
ITS ALL ABOUT what
you do with
THE DATA
The Survey Accountability Loop
Step I
REDEFINE SURVEY
QUESTIONS
IDENTIFICATION OF
NEEDS AND PROGRAM
PRIORITIES
Deploy survey and obtain
feedback (ratings and
comments)
(Oct)
EVALUATION AND
RECOGNITION FOR
ACTIONS TAKEN
Step VI
Step II
Realign with strategic goals
and/or course correct
Identify themes and
opportunities
(Jul – Sept)
(Dec)
Step III
Opportunities to Action:
Develop and implement action
plans and set goals (share with
team and Sr. Leadership)
Step V
Did changes result in goal
attainment?
(Jan – Feb)
Communicate impact and share
results with Senior Leadership
(June)
Step IV
PROJECT AND PROGRAM
IMPLEMENTATION
Follow up and assess
performance and impact of
action plans
(Feb-May)
RESPONSE PLANNING,
GOAL SETTING
Staff@Work Case Study: Health Sciences Development:
We Heard You
Risk Scenario: Based on FY13 Staff@Work
survey results, Health Sciences
Development identified 12 opportunities
for improvement
Data-Driven Action Taken: Sr. Leadership
Team developed and executed
comprehensive strategies to address each
opportunity under the “We Heard You”
campaign
Outcome: FY14 survey results revealed
significant score improvements AND the
largest fundraising year for Health
Sciences in the history of UC San Diego!
Case Study: We Heard You – Areas of Opportunity
Case Study: We Heard You – Data-Driven Action
The Senior Leadership Team worked together to develop
comprehensive strategies that addressed each of the
identified areas.
These strategies were branded under the theme “We
Heard You.”
Over the past year, the Senior Leadership Team executed
each strategy while periodically reminding the entire
department that “We Heard You.”
Case Study: We Heard You - Outcomes
Case Study: We Heard You - Outcomes
Case Study: We Heard You - Outcomes
Case Study: We Heard You - Outcomes
Quote from the Sr. Executive Director of Health Sciences
Development:
“While we know our work is not complete, the Staff@Work
survey has provided a roadmap to help guide our entire
team toward improved results, least of which is
accomplishing the largest fundraising year for Health
Sciences in the history of UC San Diego!”
ITS ALL ABOUT using
THE DATA to make a
positive impact
Examples of actions taken as a result of staff and customer survey data:
 Creation of a Professional Development and Training Program in Business and Financial Services which has resulted in
career advancement and salary increases of an average or 21% for participants, 8 graduate level degrees and 20
professional certifications
 Dining enhanced their menu choices to include healthier and vegan options
 Housing improved lounge and shared living spaces in response to student feedback
 Facilities Management instituted a client response system to more quickly address customer requests
 Transportation offered specific commuting alternatives per the feedback received
 Campus Shuttle brought back a shuttle route after hearing the feedback from customers
 Equipment Management created a new inventory process to alleviate the burden on departments and resulted in
successful inventory of approximately 80 campus buildings and 6,000 pieces of equipment with minimal intrusion into
research or operational processes
 Procurements created a Department Outreach program to address the specific needs of targeted customers
 Career Services Center updated their Port Triton system to make their search feature more user friendly for students
seeking quality internships
 The BFS STRIVE Leadership Development Program was created and implemented to support development of highpotential employees in the department through mentorship. The program increased diversity, spurred career growth,
encouraged professional development and fostered mentorships for the participants. It is now identified as a
University “best practice” in succession planning
For more examples, quotes from leaders, and impact, visit http://blink.ucsd.edu/sponsor/OSI/opa/index.html.
ITS ALL ABOUT THE DATA
so you can make meaningful
interpretations of what is important to
people based on their attitudes and
feelings, and take actions to make a
positive impact on the mission of your
university
Back to our collaboration
with CSU…
The timeline for launching CSU Chancellor’s Office
Business and Finance Staff@Work Survey
Approximate Pre-Launch Time: 8-10 weeks
Sponsor
meeting
(CSU and
UCSD)
4 hours
4/15/15
Project approval,
internal
communications
rollout (CSU)
8 weeks
Working
session
and
official
kickoff
(CSU and
UCSD)
4 hours
6/15/15
Develop hierarchy,
email lists,
marketing (CSU)
Deploy survey
(UCSD)
Analyze results
(UCSD)
UCSD Delivers
results to CSU
Build customized
survey questions
and application,
testing (UCSD)
6 weeks
2 weeks
4 weeks
4 weeks
8/31 survey open
8/25
Survey announcement
10/2
survey closed
10/15/15
initial results
11/15/15
final reports
CSU Chancellor’s Office plan
• Deploy the Customer Satisfaction Survey to customers in the
Chancellor’s Office and 23 CSU campuses
• Identify metrics for the other two perspectives of the BSC
• Financial/Stakeholder
• Internal Business Processes
• Now have benchmark capability with UCSD on the same survey
questions year over year to collaborate and partner on best practices
Appendix:
Example screenshots of the
Staff@Work Survey and Faculty and
Staff Customer Satisfaction Survey
Questions?
Contacts:
Alexis Naiknimbalkar, anaik@calstate.edu
(to learn more about how CSU has implemented the surveys at the
Chancellor’s Office!)
Angela Song, aysong@ucsd.edu
(to learn more about the surveys and how to bring them to your campus!)
Some screenshots of the Staff@Work Survey
Instrument
Welcome Page
English or
Spanish
Know what you are
rating & where you
are in the survey
Your department
name pops up if
you hover over
Anonymous
comments
Highlight a colleague or
leader that’s made a
difference!
Conduct and behavioral
questions for VC Equity,
Diversity, Inclusion
initiatives
Choose to “opt out” from
further reminders
Thank you gift for participation!
Some screenshots of the Faculty and Staff Survey
Instrument
Key words
for search!
Searchable by VC area,
service category, or
alphabetically
Save and finish
later
Navigation
Special recognition for a
department
Once you submit your survey,
you cannot change responses.
You are automatically returned to
this page. You can also come
back here to print you coupon.
You can click out of
the survey to our
webpage to see
recent Actions Taken
and other information
This is the coupon that everyone receives for a
gift item at the Bookstore. Each coupon is
barcoded and unique
This is the promotional banner that will be posted in three different locations on campus
Email to our VC contacts and departments in preparation for the launch…
The communication “kit” we provided departments to help encourage response participation
Note: reminders are
only sent to people
who have not
submitted their survey