Thursday, August 21, 2014

A Chart is not worth 1,000 words

Let’s say someone presents the following graph as “definitive proof that there is no connection between spending and achievement.”  Would you believe them?  Would you look at the lines on the chart and agree that the horizontal lines showing NAEP performance and the lines moving up the page showing per-pupil spending are enough to conclude the two things are not related?


After reading this blog post, I am hoping your answer will be “no.”


KPI_Chart001.jpg

In his book “How to Lie with Charts,” Gerald Everett Jones states the following:


“All charts and graphs are forms of data reduction, or summary.  A summary gives an overall picture, or general shape, of the underlying detail, or source data.  So, a summary can be useful for highlighting a valid trend or can be misleading by obscuring exceptional results that might be significant.”


My goal with this blog post is to use the graph presented above to explore some issues around how data is presented, and what to consider when viewing data presented visually in this manner. Let’s start by talking about the data elements presented.


Jones says, “Inadvertently leaving off helpful labels is the quickest way to perjure yourself.  Be sure to document sources of information as chart notes.”  The chart above is guilty of having some pretty vague source notes that leave a lot of questions.


For NAEP, the chart presents 4th grade reading scores.  Why was this particular statistic chosen? From the NAEP site, these are some of the possible statistics available that could have been used along with the Kansas values for 2005 and 2013:

Statistic
Grade Subject 2005 2013
Average Score 4th Reading 220 223
Average Score 4th Math 246 246
Average Score 8th Reading 284 290
Average Score 8th Math 267 267
Percent at Basic or Above 4th Reading 88 89
Percent at Basic or Above 4th Math 66 71
Percent at Basic or Above 8th Reading 77 79
Percent at Basic or Above 8th Math 78 78
Percent at Proficient or Above 4th Reading 47 48
Percent at Proficient or Above 4th Math 32 38
Percent at Proficient or Above 8th Reading 34 40
Percent at Proficient or Above 8th Math 35 36

As can be seen, some of these comparisons show no change (4th math scores, 8th math scores, 8th math percent basic); some show slight improvement (4th reading percent basic, 4th reading percent proficient, 8th math percent proficient); and some show larger improvement (4th reading scores, 8th reading scores, 4th math percent basic, 8th reading percent basic, 4th math percent proficient, 8th reading percent proficient).


In addition, there is the option of looking at state rankings based on the above, plus the choice of looking at all students, free and reduced-price lunch eligible students, free and reduced-price lunch ineligible students, and a variety of other divisions.


But if the goal is to show overall achievement changes, what if you calculated a composite score across the subjects and grades? Correlation analysis suggests that performance on these two subjects across two grade levels are highly related, as the following table demonstrates.  


Correlations for Combined and Individual Assessments, 
Percent Basic & Proficient, All Students, NSLP Eligible and Ineligible

Test
All
Basic
All
Proficient
NSLP Eligible
Basic
NSLP Eligible
Proficient
4th Math 0.95 0.96 0.93 0.94
4th Reading 0.94 0.94 0.9 0.87
8th Math 0.96 0.96 0.93 0.92
8th Reading 0.96 0.95 0.92 0.9

Though some “noise” is introduced into the comparisons when multiple measures are averaged in this fashion, the highly correlated nature of the measures being combined suggests the impact of this noise should be minimal.


So what do the composite scores look like?

Statistic
Grade Subject 2005 2013
Average Composite Composite 254 257
Percent at Basic or Above Composite Composite 77 79
Percent at Proficient or Above Composite Composite 37 40

This data would suggest that over time, outcomes as represented by NAEP scores have improved for Kansas students between 2005 and 2013.  


The point being that there are many ways to look at NAEP scores, and using different statistics yields apparently different interpretations of whether outcomes improved across time or not.  To take an exam of one subject at one grade level and to say it represents overall student achievement is called "overgeneralization."


Plus, NAEP is not the only measure for outcomes.  There are other assessments, such as those administered by the state, ACT, and SAT.  Then there are graduation rates, percent of students needing college remediation, and many of other measures of student outcomes.


Next the chart shows “Per Pupil Spending,” both in actual dollars and adjusted for inflation.  


There is no indication of what data element from KSDE’s data was used to represent “per pupil spending” on this chart.  The notes at the bottom of the chart indicate “KPERS added to total aid for 1993 to 2003.”  So, “per pupil spending” is represented by “total aid,” but again there is no definition given for “total aid.”  Further, why was KPERS added for these years and not the others?  What would the lines have looked like if this had not been added?  What other financial data might have been included in this figure?  


Investigation shows that starting in 2004, a law was passed to pass KPERS amounts through school district budgets. This is a reasonable justification for adding KPERS values to the earlier years, but additional research was required before that fact was apparent.


The chart gives no explanation for how dollars were adjusted for inflation.  Given the fact that the two lines converge at 2013, we can assume that the line is supposed to represent the amounts in 2013 dollars, but what calculation for inflation was used?  The Consumer Price Index, or some other calculation?  Was it a national or regional index figure?  


So again, there are a lot of variables that could be presented to represent spending compared to outcome data, and the choice of these variables impacts the story that is told.


Moving on from here, let’s talk about the chart itself.  


Looking at the horizontal, or “y” access, we see that the scale is supposed to represent years. Are these calendar, fiscal, or school years?  Knowing how the data is reported, I can tell you the data is provided in terms of the fiscal year for the financial data and school year for the NAEP scores. The fiscal and school years both typically run from July 1st to June 30th, so it is reasonable to look at them both on the same horizontal scale.


However, note that the values are as follows:  ‘98, ‘02, ‘03, ‘05, ‘07, ‘09, ‘11, ‘13.  There are four years between the first two values, one year between that and the next, then two years between each of the rest. But the values are presented evenly spaced along the horizontal axis.  This distorts the actual trends because you are looking at longitudinal data on an inconsistent scale.


Plus, though NAEP data is only available for every other year, the financial data is available for every year in the chart.  Were values for each year used to draw the line for the financial data, or only values from the years where NAEP data was available?  


As for the vertical scale, the biggest issue is that two different "y" axes are presented on the same graph. Jones indicates in his book that there is nothing inherently dishonest about presenting data this way, but you have to be cautious about the scales you use for each axes to ensure you are comparing accurately.  


One possible alternative to presenting data with multiple "y" axes would be to report both NAEP scores and funding in terms of percent change from the previous period.  This would allow for all four lines to follow the same "y" axis.  


Otherwise, you must be very cautious about the scales used.  Jones says:


“The impression given by an xy or radar chart can be changed dramatically by manipulating the scales of its axes.  You can flatten or exaggerate the spikes in a curve by scaling axis values up or down, or by expanding or contracting the range of axis values.”


The NAEP scores are calculated on a scale of 0 to 500, as our example scale shows.  However, when you look at the average scores across states, you see that the range of actual scores is pretty restricted.  For example, in 2013 the range of average 4th grade reading scores by state is 206 to 232.  Restricting the scale to the minimum and maximum observed scores would have shown more dramatically the increase in scores over time for Kansas, and also shown that the Kansas average score was in the upper half of states’ average scores.


The financial data is typically shown on a scale from 0 to somewhere just above the maximum value, so the example chart’s presentation of the per pupil funding is appropriate in this regard.


In conclusion, I hope this blog post has given you some things to think about the next time you are presented with a chart and told it shows you “the truth.”  Make sure you look for the questions that need to be asked and ask them.


In terms of the actual data being presented, KASB will soon be releasing an analysis of the relationship between spending and outcomes in Kansas, and we will do our best to present the data as accurately and completely as we can.  In the meantime, I wanted to share a chart KASB prepared last year that reviews similar data to the example provided above.  As you can see, the picture it paints is very different from the one at the beginning of this post.





Thursday, August 14, 2014

Response to “Taxpayer-focused perspective on K-12 staffing” from CJ Online 2014-08-10

In this post, we will respond to some of the assertions/statements made in a Topeka Capital Journal blog entry from Sunday, August 10th entitled “Taxpayer-focused perspective on K-12 staffing.”


Below are a few quotes from the blog that will be addressed specifically in order to provide additional information and considerations.  Note that the data used for this response can be found here.


  • “Most districts (234 of 286) do not employ Assistant Superintendents, which would indicate that the positions may not be a necessary expense for which all citizens are obligated to fund.”


First, it is important to note that the funding for a particular school/district would not change if the district decided to hire or fire an assistant superintendent.  The blog post is talking about allocation of existing resources; which speaks to efficiency but not to overall funding.


Second, note that there are no standard naming conventions for administrative staff within districts.  The certified staff list includes 26 different job titles and the non-certified includes 28, but the manual for completing these forms from KSDE does not include definitions for each position.  Here are the titles and frequencies from the Certified and Non-Certified counts that the blog post includes as "managers":
Title - Certified
Total FTEs
# of Districts
Superintendent
253.8
286
Assoc./Asst. Superintendent
76.6
52
All Other Directors/ Supervisors
179.9
87
Directors/ Supervisors Spec. Ed.
117.4
53
Instructional Coordinators/ Supervisors
156.3
47
Directors/ Supervisors Career/ Tech Ed
19.4
18
Directors/ Supervisors of Health
9.0
9
Title - Non-Certified
Total FTEs
# of Districts
Business Managers
112.1
115
Assistant Superintendents
8
5
Business Services:Directors/ Coordinators/ Supervisors
99.6
57
Food Service:Directors/Coordinators/Supervisors
293.7
224
Maintenance and Operation:Directors/ Coordinators/ Supervisors
411.3
227
Other:Directors/ Coordinators/ Supervisors
173.4
49
Technology:Directors/ Coordinators/ Supervisors
220.1
195
Transportation:Directors/ Coordinators/ Supervisors
157.9
188


It is entirely possible that what one district calls a Director of Career and Tech Ed might be called the Associate Superintendent for Tech Ed in another district.  The title alone does not indicate what duties each staff person actually performs, or how similar positions with different titles can be from one district to the next.  


The blog post cites Turner-KC as having 6 Assistant Superintendents.  First of all, that number is incorrect; the file actually shows 3 Assoc./Asst. Superintendents for USD 202 in the Certified table and 3 in the Non-Certified table; indicating the Assistant Superintendents were mistakenly repeated across tables.  KSDE confirmed this after a conversation with USD 202 admin staff, who have been instructed on how to report this data correctly in the future.


In addition, of the titles listed in the table above, Turner does not have many manager positions beyond these assistant superintendents.  Other districts have staff that perform much the same functions as the three assistants at Turner, but are called something else.  KASB collects similar information from school districts related to administrators.  According to KASB’s information for the 2012-13 year, Turner had an “Assistant Superintendent for Instruction/Curriculum” and an “Assistant Superintendent for Personnel;”  showing that two of the positions they reported to KSDE as “Assistant Superintendents” were very similar to other director/manager positions at other districts as shown on the list above.


Finally, we all should understand that no two districts are created equal; they are made up of different students, different communities, different sizes, and each have unique needs.  Dictating what titles staff at each of these districts can have is not a road to improved efficiency.  After all, what’s in a name?  


  • “...the average district had 153 full time equivalent students (FTE) enrolled for every manager.”


The blog post does not specify where the FTE enrollment numbers came from.  KSDE reports enrollment in terms of Headcounts on the same K-12 reports page where the Certified and Non-Certified numbers are publicly available, so it is unclear why these numbers were not used. Nonetheless for consistency we will use FTE enrollment numbers from the Comparative Performance & Fiscal System.


Using District Total FTE, and including the titles from the table above as “Managers,” the average ratio across districts is 153.87 student FTEs to every 1.0 FTE for manager positions.  This is close to the value of 153 the blog post cited for the average district.  However, calculating an average of the student to manager ratios across district means that district size is not taken into account.  If you instead total all students in the state and divide them by all “managers” across districts in the state, you see a ratio of 201.82 student FTEs to every 1.0 FTE for manager positions.  


The difference between these two calculations suggests that district size has an impact on the ratio of managers to students.  District size can be examined in two ways; in terms of number of students, and in terms of number of buildings.


Size - Student Population


If we look at the average ratio of student FTEs to manager FTEs based on total student population, we see the following:


Student FTEs
Number of Districts
Average Ratio
Greater than 10,000
7
172.71
1,000 to 10,000
73
130.04
500 to 1,000
71
86.53
250 to 500
85
72.83
100 to 250
44
50.34
Less than 100
6
31.79




This data suggests that the average ratio of student FTEs to manager FTEs increases as district size increases.  This implies an economy of scale, with larger districts showing fewer managers per student than smaller districts.


However, it is important to note that in smaller districts it is very common for the managers to also serve other roles; from teacher to coach to bus driver to any number of other non-managerial functions.  This is less common in larger districts where more positions make it more possible to have clearer divisions of labor.  So, the lower ratio in smaller districts can be somewhat misleading, as these managers are also performing non-managerial duties that would be separate positions in larger districts.


Also, it is important to note that some districts house Special Education Cooperatives; which require additional staff often at the manager level.  These are not easily identified and controlled for in the Certified and Non-Certified personnel report, and can have an adverse effect on the ratios reported for any district that supports a cooperative.  Further, the larger the district, the more likely they are to be a host for a Special Education Cooperative, so it is likely that the decrease in the student manager ratio for the districts larger than 10,000 student FTEs is at least in part due to the other services they support; such as those provided by the Special Education Cooperatives.


Size - Number of Buildings


Looking at the number of managers in a district compared to the number of building is an alternative to looking at the ratio between student and manager FTEs.  Looking from a business perspective, you would expect to have more managers if you have more locations, so it stands to reason that the number of managers would increase with the number of buildings, but that overall the ratio would remain fairly consistent.  


Using the number of buildings per district as reported by KSDE, we find that the average ratio of Number of Buildings to Manager FTEs is 1.22, and the statewide total ratio is 0.77.  Breaking this down into groups by the number of buildings, we see the following:


Buildings
Districts
Ratio of Buildings to Manager FTEs
Ratio of Student FTEs to Manager FTEs
> 20
18
0.56
167.23
16-20
15
0.70
146.49
13-15
21
1.07
99.84
10-12
45
1.13
106.47
7-9
142
1.38
73.90
1-6
45
1.34
63.41


Note that all buildings reported to KSDE are included in these calculation; not just traditional elementary, middle/junior, and high schools.  






As the table and charts show, the general trend is for the ratio of buildings to managers to decrease as the overall number of buildings increases, while the the ratio of students to managers increases as the overall number of buildings increases.  So, the bigger the district, the fewer buildings a manager is responsible for, but at the same time the bigger the district, the more students each manager is responsible for.    Since overall student population is closely tied to the number of buildings per district (correlation = .96), it is difficult to determine how much these relationships are due specifically to the number of buildings, or to district size in general.  


  • “USD 322 Onaga-Havensville-Wheaton in Pottawatomie County is the most efficient in that regard, with 608 students-per-manager.”


USD 322 has a Superintendent who also serves as a Principal, which is why they are reported for only .5 FTE; resulting in a ratio of 608 when their headcount only equals 304.  Because it is common for superintendents to also serve as principals in smaller districts, presenting a ratio of managers to students that excludes principals is problematic.


Again using District Total FTEs, and this time including principals and assistant principals, the average ratio across districts is 83.67 student FTEs to every 1.0 FTE for manager positions.  The total ratio across districts is 114.23 students to every 1.0 FTE for manager positions.  With this calculation, USD 322 shows a ratio of 152 student FTEs  per 1.0 FTE for manager positions; still slightly better than the average, but not to the absurd extent suggested by the calculation presented in the CJ Online blog post.


  • “If a small district with similar composition of economically disadvantaged students can operate with few managers and have comparable outcomes, what does that say about districts with large management structures?”


Based on the data for Kansas, the number of managers in a district has no measurable impact on assessment results.  Regression analysis was performed with the district student-manager ratios (including principals) as the independent variable and the average percent meeting standards or above on the state assessments as the dependent variable.  Results indicate there is no relationship between these two measures (p > .6, correlation = .03, percent at basic and above on KS state assessments averaged across reading and math for grades 3-8 and 11).  


Does that mean the number of managers a district has doesn’t matter?  No, it does not mean that, but what it does mean is that a wide variety of other factors impact student achievement to the point that mathematically the number of managers per students cannot be used to predict student achievement.  These other factors include student teacher ratios, teacher quality, administrator quality, parental involvement, spending, and any number of others.  


  • “...more examples of significant variances in management and support staffing”


As noted above, there is wide variation the titles used for staff position and (particularly in smaller districts) it is common for a single staff person to “wear many hats.”  Focusing on the titles and their presence or absence in districts misses the point.  However, you will note that most of the specialized positions the blog post calls out on the bulleted lists are within larger districts, which as we discussed earlier are more likely to be able to have staff who can specialize in specific areas whereas staff in smaller districts are more likely to have to serve multiple roles.

In conclusion, we all agree that Kansas Schools should be empowered to run as efficiently as possible in order to maximize the value of the tax dollars going into the system and to maximize the success rates for the students coming out of it.  However, this is not going to be accomplished by following recommendations based on misinterpreted data in order to target changes that would likely have no impact on student outcomes.