Tuesday, July 28, 2015

Being Like the Cool Kids




In a previous blog post, I discussed KASB’s efforts to identify peer states; those states that are similar to Kansas.  In this post, I will discuss KASB’s method for identifying states that outperform Kansas, which we will call “Aspiration States.”


Looking at states that perform better than Kansas on a majority of outcome measures might help us identify characteristics, such as funding formula components, that are associated with better student outcomes.


The process for identifying aspiration states is similar to the process used for identifying peer states using state values on a variety of measures.  We looked for states outperforming Kansas on at least 7 of the following 14 achievement and attainment measures:


  • Student Attainment
    • Freshman Graduation Rate
    • Cohort Graduation Rate (4 measures)
      • All Students
      • Economically Disadvantaged Students
      • Limited English Proficiency Students
      • Students with Disabilities
    • Percent of 18-25 year olds with a high school diploma
  • Student Achievement
    • Percent performing at or above “Basic” on the NAEP assessment (3 measures)
      • All Students
      • Students Eligible for the National School Lunch Program
      • Students Not Eligible for the National School Lunch Program
    • Percent performing at or above “Proficient” on the NAEP assessment (3 measures)
      • All Students
      • Students Eligible for the National School Lunch Program
      • Students Not Eligible for the National School Lunch Program
    • Percent meeting all four ACT benchmarks (adjusted for percent participation)
    • Average composite SAT score (adjusted for percent participation)


The following is the list of the identified Aspiration States, along with information on how they compare to Kansas on spending and student outcomes.  As noted in the post on Peer States, “similar to Kansas” on student demographics or population characteristics is defined as within plus or minus one half a standard deviation of Kansas’s value for the measure(s) in question.  


Aspiration States (better than KS on at least 8 out of 14 outcome measures)
  • New Hampshire (10)
    • Spends more on 6/6 funding measures
    • Has fewer students per district and school, but more students per staff
    • Similar to Kansas on 0/5 student demographic measures
    • Similar to Kansas on 1/5 population measures
  • New Jersey (10)
    • Spends more on 6/6 funding measures
    • Has fewer students per staff, but more students per district and school
    • Similar to Kansas on 0/5 student demographic measures
    • Similar to Kansas on 0/5 population measures
  • Massachusetts (9)
    • Spends more on 6/6 funding measures
    • Has more students per district, school, and staff
    • Similar to Kansas on 2/5 student demographic measures
    • Similar to Kansas on 1/5 population measures
  • Vermont (9)
    • Spends more on 6/6 funding measures
    • Has fewer students per district, school, and staff
    • Similar to Kansas on 0/5 student demographic measures
    • Similar to Kansas on 2/5 population measures
  • Minnesota (8)
    • Spends more on 6/6 funding measures
    • Has fewer students per school, but more students per district and staff
    • Similar to Kansas on 3/5 student demographic measures
    • Similar to Kansas on 2/5 population measures



NH
NJ
MA
VT
MN
Freshman Grad Rate
Cohort Graduate Rate
CGR - FRL
CGR - ELL
CGR - IDEA
Pct 18-25 - High School
NAEP Pct Basic
NAEP Pct Proficient
NAEP Pct Basic NSLP Eligible
NAEP Pct Proficient NSLP Eligible
NAEP Pct Basic NSLP Ineligible
NAEP Pct Proficient NSLP Ineligible
Pct Meeting All 4 ACT Benchmarks Adjusted
SAT Mean Score Adjusted


↑ = Value Higher than Kansas
↓ = Value Lower than Kansas


So, what does this tell us?  


First, only five states perform better than Kansas on a majority of the outcome measures examined.  
Second, all of the five aspiration states spend more per pupil than Kansas on all six funding measures.  


Third, there is a fairly even mix of states that have more students per district, school, and staff and those that have fewer students per district, school, and staff.  3/5 states had more students per district, 2/5 had more students per school, and 3/5 had more students per staff.


Fourth, none of these states were identified previously as a “peer state.”  Further, only one state is similar to Kansas on a majority of the student demographic measures (Minnesota), and no state is similar to Kansas on a majority of the population characteristics measures.   It is important to remember that these states differ from Kansas in student demographic and population characteristics, which are not under the control of the state education system.   


Next time we will talk about controlling for differences among states related to student demographics and population characteristics and how such an analysis might better inform us in terms of which states we should be looking to for ideas.  

To examine the data more closely, use the following links:

Wednesday, July 22, 2015

Knowing Your Peers Revisited


Last July, we attempted to identify “peer” states for Kansas, as described in this blog post.  We used four variables, Z scores, and standard deviations in order to try and determine which states are most like Kansas. Here is the resulting list of states:


  • Arizona
  • Arkansas
  • Idaho
  • Iowa
  • Minnesota
  • Nebraska
  • Oklahoma
  • Oregon


The four variables we used were:


  • Population per Square Mile
  • Percent of Students Eligible for Free or Reduced-Price Lunch
  • Percent of Students Participating in Programs for English language Learners
  • Percent of Students Served under IDEA (Special Education)


We selected these variables because they are commonly cited as factors influencing the nature of education at the state level.


Since July, we have spent a lot of time with this state-level data, and even undertook a large multi-year multiple regression analysis trying to determine what factors, in what combinations, have the largest impact on student outcomes. The results of the analysis suggested a much larger list of variables had a noticeable impact on student outcomes than the ones we had used to identify peer states.


In January, KASB released our first “Kansas Educational Achievement Report Card,” in which we discuss Kansas’ ranking on a variety of funding, outcome, and demographic variables to determine how our state compares with others.  


Based on the work we’ve done since last July, we decided it was time to revisit the notion of peer states and see if we can better identify those states that are similar to us. Taking the multiple regression analysis and the report card data, we identified 34 measures to be included.  For each of these variables, we pulled the most recent year’s data.  From here we could have calculated some form of overall rank or some kind of average of ranks to attempt to identify states similar to Kansas.  


But when we say we want to compare ourselves to states similar to ourselves, what are we really going for?  And further, is it reasonable to expect that we would want to compare ourselves to the same states each time regardless of the kinds of questions we are asking?  Thinking about this, and looking at how we are going to organize the next version of the Kansas Education Achievement Report Card, we put the 34 variables into six categories:


  • Student Attainment
    • Freshman Graduation Rate
    • Cohort Graduation Rate (4 measures)
      • All Students
      • Economically Disadvantaged Students
      • Limited English Proficiency Students
      • Students with Disabilities
    • Percent of 18-25 year olds with a high school diploma
  • Student Achievement
    • Percent performing at or above “Basic” on the NAEP assessment (3 measures)
      • All Students
      • Students Eligible for the National School Lunch Program
      • Students Not Eligible for the National School Lunch Program
    • Percent performing at or above “Proficient” on the NAEP assessment (3 measures)
      • All Students
      • Students Eligible for the National School Lunch Program
      • Students Not Eligible for the National School Lunch Program
    • Percent meeting all four ACT benchmarks (adjusted for percent participation)
    • Average composite SAT score (adjusted for percent participation)
  • Money (School Spending)
    • Total Revenue Per Pupil (2 measures)
      • Actual Dollars
      • Dollars Adjusted for Regional Cost of Living (RPP)
    • Current Spending Per Pupil (2 measures)
      • Actual Dollars
      • Dollars Adjusted for Regional Cost of Living (RPP)
    • Spending on Instruction Per Pupil (2 measures)
      • Actual Dollars
      • Dollars Adjusted for Regional Cost of Living (RPP)
  • Organization Size
    • Ratio of Students per District
    • Ratio of Students per School
    • Ratio of Students per Staff Member
  • Student Demographics
    • Percent of Children at 100 percent Poverty
    • Percent Eligible for Free or Reduced-Price Lunch
    • Percent Served under IDEA (Special Education)
    • Percent in English Language Learner Programs
    • Percent of Students who are Non-White
  • Population
    • Median Household Income
    • Population per Square Mile
    • Percent of 25 Year Olds and Older with a High School Diploma
    • Percent of 25 Year Olds and Older with a Bachelor’s Degree
    • Percent of 25 Year Olds and Older with a Graduate Degree


By grouping the variables this way, we can ask more specific questions, such as “How do other states who are similar to Kansas in terms of school spending do on the ACT exam?” Because we are usually interested in comparing ourselves in terms of something specific, these kinds of groupings might help us more than the generic groupings we’ve been discussing.


Let’s assume, for a moment, that when we say “peer states,” we are talking about states that have similar populations.  We’d want to compare Kansas to other states based on the measures in the “Population” category, for sure.  But more specifically, we would want to look at other states that have similar public school student populations, so we’d also want to compare on the measures in the “Student Demographics” category.  


Combining these two categories, we have 10 demographic measures.  We took these measures, and marked any state within plus or minus one-half of a standard deviation of Kansas’s value as “similar to Kansas.”   Using this approach, we found the following number of “similar” states for each measure:


  • Student Demographics
    • Percent of Children at 100 percent Poverty - 14
    • Percent Eligible for Free or Reduced-Price Lunch - 14
    • Percent Served under IDEA (Special Education) - 21
    • Percent in English Language Learner Programs - 11
    • Percent of Students who are Non-White - 15
  • Population
    • Median Household Income - 18
    • Population per Square Mile - 31
    • Percent of 25 Year Olds and Older with a High School Diploma - 16
    • Percent of 25 Year Olds and Older with a Bachelor’s Degree - 15
    • Percent of 25 Year Olds and Older with a Graduate Degree - 14


The following shows the states that were similar to Kansas based on the criteria above on at least 6 of the 10 demographic measures, along with information on how they compare to Kansas on spending, student outcomes, and student ratios:


Peer States (similar (+/-.5 Std Dev) to KS on at least 6 of 10 demographic measures)
  • Oregon (7)
    • Spends more on 0/6 funding measures
    • Better on 1/14 outcome measure
    • Has more students per district, school, and staff
  • Washington (7)
    • Spends more on 0/6 funding measures
    • Better on 4/4 outcome measures
    • Has more students per district, school, and staff
  • Illinois (6)
    • Spends more on 6/6 funding measures
    • Better on 2/14 outcome measures
    • Has more students per district, school, and staff
  • Michigan (6)
    • Spends more on 6/6 funding measures
    • Better on 2/14 outcome measures
    • Has more students per district, school, and staff
  • Nebraska (6)
    • Spends more on 6/6 funding measures
    • Better on 5/14 outcome measures
    • Has fewer students per district, school, and staff
  • Pennsylvania (6)
    • Spends more on 6/6 funding measures
    • Better on 4/14 outcome measures
    • Has more students per district, school, and staff
  • Wisconsin (6)
    • Spends more on 6/6 funding measures
    • Better on 5/14 outcome measures
    • Has more students per school and staff, but fewer students per district



OR
WA
IL
MI
NE
PA
WI
Pct of Children in Poverty (100%)
Pct of students eligible for FRL
Pct of students (3-21) Served Under IDEA
Pct of students in ELL programs
Pct of Public School Students, Non-White
Median Household Income - Actual Dollars
Population Per Square Mile
Pct of 25-year-olds and > - HS completion or >
Pct of 25-year-olds and > - Bachelors or >
Pct of 25-year-olds and > - Graduate degree or >


↑ = Value Higher than Kansas
↕ = Value Similar to Kansas
↓ = Value Lower than Kansas


So, what does this tell us?  


First of all, of the 7 states similar to us, 5 spend more than Kansas per pupil based on all 6 funding measures.  That is 70% of our peer states.


Second, of the 7 states similar to us, no state has better student outcomes on a majority of achievement and attainment measures.  Peer states perform better on Kansas on anywhere from 1 to 5 of the measures each; meaning Kansas outperforms each of these peer states on anywhere from 65% to 90% of the achievement and attainment measures.


Third, Kansas has fewer students per district than 5/7 peer states, fewer students per school than 6/7 peer states, and fewer students per staff than 6/7 peer states.   It is unclear specifically how these ratios impact student outcomes, but it is notable that Kansas has more districts, schools, and staff members per student than most of its peers.


Fourth, it is important to note that many of our neighbor states; including Colorado, Iowa, Missouri, and Oklahoma, are not on this list of peer states.  Kansas is frequently compared to our neighbor states, despite the fact that we are notably different from them in terms of student demographics and population characteristics.


Next time, we’ll look at states that outperform Kansas on a majority of outcome measures and discuss what we might be able to learn from them.  

To examine the data more closely, use the following links: