From a Culture of Evidence to
a Culture of Action & Inquiry
APPENDIX:
Examples of Outcomes Data
That Matter
Dr. Rob Johnstone
Aspen / ATD Leadership Symposium
Wye River, MD
April 1, 2014
www.inquiry2improvement.com
Acknowledgements
 The content in this presentation has been collected from a
variety of sources, all of which are publicly available on
websites or have been presented in public settings.
 These sources include local college IR efforts, state system
efforts, and national projects such as:
 Completion by Design
 The Aspen Prize for Community College Excellence
 The Carnegie Foundation’s Statway / Quantway
 Collegemeasures.org
 Content has also been developed by and with a host of
national partners, including CCRC, the Aspen Institute, the RP
Group (CA), and AIR (American Institutes for Research)
National Center for Inquiry & Improvement
www.inquiry2improvement.com
April 2014
Examples of Data that
Matter
www.inquiry2improvement.com
3
Aspen’s Take on Data that
Matters:
Progress / Completion Outcomes Data
Labor Market Outcomes Data
Learning Outcomes Data
Equitable Outcomes Data
National Center for Inquiry & Improvement
www.inquiry2improvement.com
4
April 2014
Progress / Completion Data:
A “Quick” Look
www.inquiry2improvement.com
5
Completion Example #1
CUNY ASAP Program
Graduation Rates
National Center for Inquiry & Improvement
www.inquiry2improvement.com
6
ASAP Program Results
http://www.cuny.edu/academics/programs/notable/asap/about.html
7
Completion Example #2
Completion Rates in
Completion by Design
National Center for Inquiry & Improvement
www.inquiry2improvement.com
8
Example from CBD:
Outcomes for LAS Concentrators
Outcomes by CTE Concentrators
Completion Example #3
Valencia College
National Center for Inquiry & Improvement
www.inquiry2improvement.com
11
Outcomes of Prize finalist colleges:
Exceptional improvements over time in completion
Progress Example 1:
Grades in Sequenced Courses
National Center for Inquiry & Improvement
www.inquiry2improvement.com
13
Grades in Sequenced Courses Backdrop
• Course sequences analyzed in Mathematics,
English, Biology, Chemistry, Physics & Economics
• Relationship between grade in 1st course in
sequence and success in 2nd course in sequence
was examined.
• Enrollment data tracked over five-year period
• Selected highlight slides follow; full PPT
provided on request
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Data Too Often Presented Like This…
Count
% of Total N
Success Rate in
Math 310
A in Math 308
5,659
10%
81%
B in Math 308
7,726
13%
60%
C in Math 308
7,234
12%
37%
Unclear Status in Math 308*
1,675
3%
16%
Math 310 Repeater / Non Success
14,568
25%
41%
Math 310 Attempt was First Math Course
16,812
29%
57%
Taking Math 308 / 310 in Same Semester
4,505
8%
68%
Other
487
1%
44%
Total
58,666
100%
53%
Status in Math 310
National Center for Inquiry & Improvement
www.inquiry2improvement.com
15
And Not Often Enough Like This…
Pre-Algebra
Introductory
Algebra
Intermediate
Algebra
TransferLevel
Math 306
Math 308
Math 310
Math 1300s
The grade received by
students in Math 308
Success Rate of those same
students in Math 310
100%
81%
A
B
90%
80%
70%
Those placing
directly into
Math 310
60%
57%
60%
50%
37%
40%
30%
20%
C
10%
0%
Calculus Course Sequence
Calculus I
Calculus II
MATH 2413
MATH 2414
The grade received by
students in Math 2413
Success Rate of those same
students in Math 2414
100%
90%
A
B
78%
Those placing
directly into Math
2414
72%
90%
80%
70%
60%
50%
50%
40%
30%
20%
C
10%
0%
Developmental Education Writing Course Sequence
Two Levels
below
Transfer
One Level
below
Transfer
One Level
Transfer
below
Level English
Transfer
ENGL 306
ENGL 307
ENGL 1301
The grade received by
students in English 307
Success Rate of those same
students in English 1301
100%
90%
A
Those placing
directly into
English 1301
74%
71%
61%
80%
70%
60%
B
46%
50%
40%
30%
20%
C
10%
0%
College Biology Course Sequence
Anatomy
Physiology
BIO 2401
BIO 2402
The grade received by
students in Biology 2401
Success Rate of those same
students in Biology 2402
100%
88%
A
B
Those placing
directly into
Biology 2402
72%
90%
80%
70%
67%
60%
50%
50%
40%
30%
20%
C
10%
0%
College Chemistry Course Sequence
General
Chemistry I
General
Chemistry II
CHEM 1411
CHEM 1412
The grade received by
students in Chemistry 1411
Success Rate of those same
students in Chemistry 1412
100%
A
B
86%
90%
80%
Those placing
directly into
Chemistry 1412
70%
66%
70%
60%
46%
50%
40%
30%
20%
C
10%
0%
Economics Course Sequence
Macro
Economics
Micro
Economics
ECON 2301
ECON 2302
The grade received by
students in Economics 2301
Success Rate of those same
students in Economics 2302
100%
89%
A
81%
Those placing
directly into
Economics 2302
74%
71%
90%
80%
70%
60%
B
50%
40%
30%
20%
C
10%
0%
Progress Example 2:
Developmental Education
Cohort Tracking
National Center for Inquiry & Improvement
www.inquiry2improvement.com
22
Math cohort tracking starting in Pre-Algebra
PRE-ALGEBRA
INTRODUCTORY
ALGEBRA
INTERMEDIATE
ALGEBRA
TRANSFER LEVEL
100%
100%
Attempt
the
Course
72%
Pass
the
Course
90%
80%
70%
60%
50%
Of the
original
cohort
43%
Attempt
the
27%
Pass
Course
the
40%
30%
Of the
original
cohort
Course
National Center for Inquiry & Improvement
20%
18%
11%
Pass
Attempt
Of the
original
cohort
www.inquiry2improvement.com
Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.
10%
5%Pass
Att
4%
0%
Math cohort tracking starting in
Introductory Algebra
PRE-ALGEBRA
INTRODUCTORY
ALGEBRA
INTERMEDIATE
ALGEBRA
TRANSFER LEVEL
100%
100%
Attempt
the
Course
64% Pass
the
Course
90%
80%
70%
60%
50%
50%
Attempt
39% Pass
the
the
Course
Course
Of the
original
cohort
40%
30%
Of the
original
cohort
25%
19%
Pass
Attempt
20%
10%
0%
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.
Math cohort tracking starting in
Intermediate Algebra
PRE-ALGEBRA
INTRODUCTORY
ALGEBRA
INTERMEDIATE
ALGEBRA
TRANSFER LEVEL
100%
100%
Attempt
the
Course
67% Pass
the
Course
90%
80%
70%
60%
50%
Of the
original
cohort
50%
Attempt
37%
Pass
the
the
Course
Course
40%
30%
20%
10%
0%
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.
English Writing cohort tracking starting
at Developmental Writing I
DEV WRITING I
DEV WRITING II
COMPOSITION I
COMPOSITION II
100%
100%
Attempt
the
Course
71% Pass
the
Course
90%
80%
70%
60%
Of the
original
cohort
53%
Attempt
45%
Pass
the
the
Course
Course
National Center for Inquiry & Improvement
50%
40%
Of the
original
cohort
33%
Attempt
26%
Pass
the
Course
www.inquiry2improvement.com
Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.
30%
20%
10%
0%
English Writing cohort tracking starting
at Developmental Writing II
DEV WRITING I
DEV WRITING II
COMPOSITION I
COMPOSITION II
100%
100%
Attempt
the
78%
Pass
Course
the
Course
90%
80%
70%
65%
Attempt
54%
Pass
the
the
Course
Course
Of the
original
cohort
60%
50%
40%
30%
20%
10%
0%
National Center for Inquiry & Improvement
www.inquiry2improvement.com
Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.
Progress Example 3:
Look for Examples from Four-Year Schools:
Retention & Progress at Georgia State
http://www.edtrust.org/higher_ed_practice_guide for more examples
National Center for Inquiry & Improvement
www.inquiry2improvement.com
28
First Year to Second Year Retention,
Georgia State University
80%
82%
81%
FALL '00
FALL '01
FALL '02
83%
FALL '03
80%
81%
82%
83%
83%
FALL '04
FALL '05
FALL '06
FALL '07
FALL '08
Retention
National Center for Inquiry & Improvement
www.inquiry2improvement.com
29
First Year Retention & Progression
Rates - Georgia State University
82%
80%
83%
81%
83%
83%
82%
81%
80%
67%
62%
49%
27%
28%
FALL '01
FALL '02
33%
35%
FALL '03
FALL '04
39%
22%
FALL '00
Retention
National Center for Inquiry & Improvement
FALL '05
FALL '06
FALL '07
FALL '08
Retained & Sophomore
www.inquiry2improvement.com
30
April 2014
Labor Market Outcomes
Data: A Quick Look
www.inquiry2improvement.com
31
Labor Market Outcomes
Example 1:
CollegeMeasures.org ESM
for Program-Level LMI Data
National Center for Inquiry & Improvement
www.inquiry2improvement.com
32
CollegeMeasures.org is one
possibility to get LMI data
●
Collaboration between the Lumina Foundation
and AIR
●
Six states currently signed on - AR, CO, FL, TN,
TX, VA
●
Any state can theoretically participate
●
Check it out:
http://www.collegemeasures.org/esm/
National Center for Inquiry & Improvement
www.inquiry2improvement.com
33
Wages of Individual Degree Programs
at an Arkansas CC
Earnings
PRECISION PRODUCTION
$45,510
NURSING
$44,988
CRIMINAL JUSTICE
$30,297
COMPUTER/IS
$27,952
LIBERAL ARTS
$24,724
BUSINESS
$24,374
ACCOUNTING
$22,903
MEDICAL OFFICE MANAGEMENT
$20,517
CHILDCARE/SUPPORT SERVICES
$18,078
$0
$10,000
National Center for Inquiry & Improvement
$20,000
$30,000
www.inquiry2improvement.com
$40,000
$50,000
34
Wage Earning Rate of Individual <1 Year
Certificate Programs at a Colorado CC
Wage Earning Rate
BUSINESS ADMIN.
51%
HEALTH AIDES
37%
CRIMINAL JUSTICE
32%
MEDICAL ASSISTING
29%
COMPUTER/IS
28%
REAL ESTATE
24%
ARCHITECTURAL ENGINEERING
23%
LAB SCIENCE/RESEARCH
20%
DIAGNOSTIC/INTERVENTION
19%
PRACTICAL NURSING
16%
0%
10%
20%
National Center for Inquiry & Improvement
30%
40%
50%
60%
70%
www.inquiry2improvement.com
80%
90%
100%
35
Wages for AA Degrees in Business / Mgmt /
Marketing in Tennessee, By College
Earnings
COLLEGE M
$39,462
COLLEGE L
$39,127
COLLEGE K
$36,848
COLLEGE J
$36,108
COLLEGE I
$35,984
COLLEGE H
$33,954
COLLEGE G
$33,044
COLLEGE F
$32,790
COLLEGE E
$31,897
COLLEGE D
$31,031
COLLEGE C
$30,487
COLLEGE B
$29,591
COLLEGE A
$27,518
$0
$10,000
National Center for Inquiry & Improvement
$20,000
$30,000
$40,000
www.inquiry2improvement.com
$50,000
36
Wage Earning Rate of Licensed
Vocational Nurses in Virginia, By College
Wage Earning Rate
COLLEGE H
67%
41%
COLLEGE G
63%
54%
COLLEGE F
63%
52%
COLLEGE E
62%
46%
COLLEGE D
60%
37%
COLLEGE C
56%
42%
COLLEGE B
48%
22%
COLLEGE A
33%
0%
10%
20%
30%
Wage Rate
National Center for Inquiry & Improvement
44%
40%
50%
60%
70%
80%
90%
100%
Above Avg Wage Rate
www.inquiry2improvement.com
37
Labor Market Outcomes
Example 2:
Aspen Prize’s Approach for
Comparing Labor Market Outcomes
National Center for Inquiry & Improvement
www.inquiry2improvement.com
38
Aspen Prize Sample Model Comparing
Labor Market Outcomes
Employment Metrics
Earnings and Wage Metrics
County-level Economic Context Metrics
Annualized
Salaries and
Wages for
Employed
2006
Graduates in
2011
Class of 2006
Relative
Wages
Class of 2011
Composite
Employment
Index
Class of 2006
Composite
Employment
Annualized
Salaries and
Wages for
2011
Employed
Graduates
2011 County
Unemployment
Rate
County
5 Year
Employ
ment
Growth
Rate
County
Community
College
239
209
$55,784
180%
$79,204
161%
76%
8.4%
7.0%
Peer College 2
176
182
$24,969
85%
$48,130
102%
80%
9.0%
-8.1%
Peer College 3
175
154
$20,304
89%
$29,475
89%
67%
7.5%
1.4%
Peer College 4
153
151
$21,637
77%
$41,894
99%
76%
9.5%
9.9%
Peer College 5
218
208
$28,353
138%
$36,830
100%
86%
4.2%
0.9%
Peer College 6
137
157
$27,533
101%
$43,973
91%
77%
8.6%
-3.0%
Peer College 7
175
170
$23,483
96%
$38,429
89%
77%
7.6%
-7.5%
Peer College 8
147
120
$22,919
68%
$34,005
76%
46%
10.4%
1.6%
Peer College 9
164
181
$40,966
177%
$58,470
151%
72%
7.5%
5.0%
184
146
$21,227
84%
$35,818
86%
72%
8.3%
-2.5%
177
168
$28,717
109%
$44,622
105%
73%
8.1%
-1.4%
Institution
Peer College 10
Peer-group
average
National Center for Inquiry & Improvement
Class of
2011
Relative
Wages
County Civilian
Workforce
Participation
Rate
www.inquiry2improvement.com
39
Lessons from the Aspen Prize for Community College Excellence
Outcomes of Prize finalist colleges:
Exceptional short-term labor market outcomes for graduates
Average salaries of recent graduates compared to the average
for all new-hires in the region (TOP THREE PERFORMERS)
$56,576
82%
above avg.
$41,548
79%
above avg.
$28,756
$20,540
40%
above avg.
Regional
average
Lake Area Technical Institute (SD)
$23,211
$31,086
Regional
average
Regional
average
Walla Walla Community College (WA)
Brazosport College (TX)
40
COLLEGE EXCELLENCE PROGRAM
Lessons from the Aspen Prize for Community College Excellence
Outcomes of Prize finalist colleges:
Exceptional long-term labor market outcomes
$77,272
Average salaries of graduates 5 years after graduation
compared to the average for all workers in the region
(TOP THREE PERFORMERS)
65%
above avg.
$63,016
$57,044
55%
above avg.
$36,803
$45,664
38%
above avg.
$46,832
Regional
average
Regional
average
Regional
average
Walla Walla Community College (WA)
Miami-Dade College (FL)
Brazosport College (TX)
41
COLLEGE EXCELLENCE PROGRAM
April 2014
Learning Outcomes Data: A
Quick Look
www.inquiry2improvement.com
42
Goals of Learning Outcomes Assessment
•An ideal Learning Outcomes Assessment
process:
 Focuses on authenticity vs. compliance
 Is driven & supported by the practitioners
tasked with improvement
 Leads to the generation of insight that is
woven into improvement efforts
 Creates campus conversations about student
success and improving outcomes
National Center for Inquiry & Improvement
www.inquiry2improvement.com
43
Learning Outcomes Assessment Thoughts
• Lots of outcomes to assess:
 Institutional Learning Outcomes
 General Education Learning Outcomes
 Program Learning Outcomes
 Course-level Learning Outcomes
• Challenging & complex – but critical
to work on generating insight and
improvement at all levels
National Center for Inquiry & Improvement
www.inquiry2improvement.com
44
Learning Outcomes Assessment Thoughts
• Not many nationally normed LOA models
aside from standardized tests (e.g. CLA &
CAP) – many local models that are good
•Accreditation focus on LOA has resulted in
a massive collection / reporting effort –
taking away from the actual use of data for
improvement…
National Center for Inquiry & Improvement
www.inquiry2improvement.com
45
Learning Outcomes Assessment Thoughts
• Check out how CCBC’s Executive Summaries at link
tell the assessment & improvement story at the
course level
 My favorite all-time story is Chem 108 – tells how an
attempt to improve outcomes actually made it worse, but
then further redesign achieved very high levels – tells that
LOA is a journey to improvement story
• CCBC also strong on LOA at GE levels through a rubric
assessment model – see link for more info
National Center for Inquiry & Improvement
www.inquiry2improvement.com
46
Learning Outcomes Examples
• Alverno College
 The gold standard example of LOA – check out the student,
community, and faculty responses in addition to the
mechanics - link
• Valencia College
 detailed program level assessments, excellent sharing of LOA
outcomes with peers – link
• Santa Barbara City College
 closing the loop by focusing on improvement plans – link
• West Kentucky Community & Technical College
 overall approach to student learning; significant gains in
reading based on LOA assessment – link
National Center for Inquiry & Improvement
www.inquiry2improvement.com
47
April 2014
Equitable Outcomes:
A Quick Look
www.inquiry2improvement.com
48
Equitable Outcomes
Example 1:
Northern Virginia CC & Transfers
to George Mason University
National Center for Inquiry & Improvement
www.inquiry2improvement.com
49
Northern Virginia Community College Students Transferring
into George Mason University 2005–2012
Source: SCHEV 2014
Equitable Outcomes
Example 2:
Statway / Quantway looks at
Program Effects by Gender & Race
National Center for Inquiry & Improvement
www.inquiry2improvement.com
51
Statway looks at Pathway thru Algebra
Success by Gender & Ethnicity
Non-Statway
55%
Statway
49%
Statway
Non-Statway
52%
14%
Non-Statway
13%
16%
45%
Female
Black
52%
47%
Male
7%
Statway
63%
14%
18%
Hispanic
Note. Data from unknown gender status were not presented; very few were unknown.
“Other” also contains data from unknown status.
White
18%
Other*
52
Gender and Race/Ethnicity
Non-Statway
67%
Statway
59%
49%
52%
49%
52%
38%
17%
12%
7%
Black
20%
42%
20%
17%
17%
9%
Hispanic
White
Female
Other*
Black
Hispanic
White
Other*
Male
Note. Data from unknown gender status were not presented; very few were unknown.
“Other” also contains data from unknown status.
53
College Level Math Success By College
100%
Triple the Success
14
17
1
Statway
50%
18
8
13
11
9
5
3
2
6
7
19
No Difference
15
Statway
0%
0%
-50%
50%
Non-Statway
52%
Non-Statway
14%
100%
54
College Level Math Success - Faculty
Statway
100%
50%
Statway
Non-Statway
52%
14%
0%
0%
50%
Non-Statway
100%
55
Equitable Outcomes
Example 3:
Aspen Prize’s Approach for Comparing
Equitable Outcomes at the College Level
National Center for Inquiry & Improvement
www.inquiry2improvement.com
56
An Aspen Prize Finalist Achieves Near
Equity in Outcomes
National Center for Inquiry & Improvement
www.inquiry2improvement.com
57
Lessons from the Aspen Prize for Community College Excellence
Outcomes of Prize finalist colleges:
Far greater equity in outcomes than the national average
Three-year completion and/or transfer rates for underrepresented
minority students compared to the national average
49%
44%
34%
National Average
Finalist Average
Top 3 Average
•
•
•
Brazosport College (TX)
Santa Barbara City College (CA)
Santa Fe College (FL)
58
COLLEGE EXCELLENCE PROGRAM
Lessons from the Aspen Prize for Community College Excellence
Outcomes of Prize finalist colleges:
Exceptional equity in access for underrepresented populations
% Difference between underrepresented minority enrollment in institution and percentage of the
population in the college’s service area (TOP THREE PERFORMERS)
20%
15%
0% means the college’s enrollment has the same proportion of underrepresented minority students as the general
population; i.e., minorities are fully represented at the college
13%
10%
6%
5%
1%
0%
-5%
CUNY Kingsborough
Community College (NYC)
Walla Walla Community
College (WA)
Broward College (FL)
59
COLLEGE EXCELLENCE PROGRAM
Find Out More
• The National Center for Inquiry & Improvement
website
www.inquiry2improvement.com
• Dr. Rob Johnstone, Founder & President
rob@inquiry2improvement.com
• CBD Inquiry Guides on Applied Inquiry & Nuances
of Completion:
http://www.inquiry2improvement.com/publicationsresources
National Center for Inquiry & Improvement
www.inquiry2improvement.com
60