Tuesday, July 29, 2014

Knowing Your Peers

As KASB reviews data for Kansas and the nation, the question arises time and time again; which states should we compare ourselves to?  Should we look at the states that are contiguous?  Should we look at states like Texas and Florida that seem to get all the media attention?  How do we determine who our peers are?

This post describes how I have established a methodology for KASB to define states that are the most like us in terms of four key education factors:
  • Population Per Square Mile:  We know that within Kansas we often feel the divide between the large, urban districts and the small, rural districts.  The ball games they are playing are very different.  The same is true at the state level.  Using the average number of people per square mile will help determine which states are on the average in the same ballpark as Kansas.
  • Percent of Students Eligible for Free or Reduced-Priced Lunch:  This is the only economic indicator directly tied to students and schools available to us.  The percent of students whose parents earn annual income within the range making them eligible speaks to the economic background of the student population an, as an extension, of the communities our schools are a part of.  
  • Percent of Students Participating in Programs for English Language Learners:  The Percent of Students receiving English Language Learner services tells us how many of our students come from homes where very little or no English is spoken. 
  • Percent of Students Served Under IDEA:  The percent of students served under the Individuals with Disabilities Education Act speaks to the number of students in our schools with unique educational needs that require extra services and creative solutions.  
These four factors have a large impact on the nature of education within our state.  As such, it stands to reason that we should be looking to other states that are dealing with similar conditions where these four "variables" are concerned.  

Looking at these four, I wanted some way to rate each State in such a way that they could be compared and contrasted.

Okay, fair warning - we're going to talk about statistics now.  But don't panic.  We will limit the discussion to Standard Deviations, the Normal Distribution, Z scores, and the Mode, and we'll just talk about them a little:
  • Standard deviations are used in connection with means (averages), and represent the average difference between each observation and the mean.  Higher standard deviations indicate wider variation in a set of observations (numbers) than lower standard deviations.  
  • The normal distribution (a.k.a. the Gaussian distribution, the normal curve, or the bell curve) is the way a set of observations is distributed randomly, and is used to predict how frequently certain observations should occur.
  • Z scores are standardized scores based on the mean and standard deviation, and are used to compare sets of observations coming from different variables.
  • The Mode is the most frequently occurring value in a set of data.  
I calculated Z scores for each variable for each year with the intent of using them to break the states into three categories for each variable.  I classified the observations as follows:
  • observations within 1/2 a standard deviation from the mean (which is a Z score of -.5 to .5) as "Average," 
  • observations more than 1/2 a standard deviation above the mean (a Z score above .5) as "High," and 
  • observations more than 1/2 a standard deviation below the mean (a Z score below -.5) as "Low."  
Assuming a normal distribution of values, we can expect about 40% of the States to fall in the "Average" category, 30% in the "High" category, and 30% in the "Low" category for each of our four variables (School Lunch, ELL, IDEA, and Population).

Once I had Z scores for each state on the four variables for each year of data from 2005 through 2012, I selected the mode for each state as their overall classification on each variable.  The categories each state fell into didn't change over time often; usually only when a state's scores hovered close to the edge of the categories (Z scores around -.5 and .5) or who were seeing increasing or decreasing trends. Nebraska, for example, was Low for School Lunch in 2005, Average in 2006 through 2011, and High in 2012, so they were classified as Average on School Lunch overall.  

Still with me?  Good.

I took the categories for each variable and combined them into what we will call a "meta-category" for each state.  This meta-category should allow us to group together states that are similar on these four key factors, and also help us understand how states differ.  For Example, Kansas is AAAL, which means Average on School Lunch, ELL, and IDEA, but low on Population; whereas Florida is HHAH, which means High on School Lunch and ELL, Average on IDEA, and High on Population.  

Without looking ahead, what states would you think are Kansas' peers?  You might be surprised.  

As it turns out, Kansas is without peers.  There is not another state with a meta category of AAAL.  Of the remaining 49 states, none of them break out the same.  Looking at the states that share three of the four categories with us, we find:
  • Arizona (AALL - Low on IDEA)
  • Arkansas (HAAL - High on School Lunch)
  • Idaho (AALL - Low on IDEA)
  • Iowa (LAAL - Low on School Lunch)
  • Minnesota (LAAL - Low on School Lunch)
  • Nebraska (AAHL - High on IDEA)
  • Oklahoma (HAAL - High on School Lunch)
  • Oregon (AHAL - High on ELL)
So, not only are we without peers, but the states who show up as our peers based on this approach are probably not the ones you'd think of.  I have been told that the states Kansas is often compared to include:
  • North Dakota (LLAL)
  • South Dakota (LLAL)
  • Minnesota (above)
  • Nebraska (above)
  • Iowa (above)
  • Colorado (LHLL)
  • Missouri (ALAA)
  • Oklahoma (above)
  • Texas (HHLA)
About half of those states do appear similar to us, but note how far off some of them are; North and South Dakota both have lower percents of School Lunch and ELL students, Colorado is lower on School Lunch and IDEA but higher on ELL, Missouri is lower on ELL but higher on Population, and Texas doesn't match Kansas in a single category.  

As most of our parents told us, it is important to think carefully about who we compare ourselves to.  Is it reasonable to expect our performance and spending to follow the same trends as the states on the second list, or are we setting ourselves up for disappointment?

The method I outline above may not be the one we ultimately use.  I will share these results with others and determine if this makes the most sense or if there is a better way.  Either way, moving forward KASB will continue to work to improve the comparisons we make so that we can produce useful and actionable data.