Monday, August 3, 2015

Leveling the Playing Field


In previous posts, we’ve talked about identifying Peer States and Aspiration States, and about comparing their demographics, funding, and outcomes to Kansas.  In this post, we’re going to talk about how to compare states on certain measures while removing the influence of others.


You may have heard people talking about “controlling for variables” or “holding variables constant” when discussing research, statistics, and analysis.  Essentially this means you are trying to remove the influence of these variables on the relationship between other variables.  


In the previous posts, we identified six categories into which our 44 state-level measures fall:
  • Student Attainment
  • Student Achievement
  • Money (School Spending)
  • Student Demographics
  • Organization Size
  • Population


Further, we’ve had some discussions about how school spending relates to student attainment or student achievement.  


However, often folks who oppose increases in school funding argue that a straight comparison of school funding to student outcomes ignores the fact that student demographics and population characteristics (among other measures) vary greatly from state to state, and further that differences in student outcomes are more likely due to these factors than to the amount of money spent.  


In order to test out these assertions, we would need a way to remove the effects of student demographic and  population measures on the student attainment and student achievement measures.  Doing so will allow us to create a list of “higher impact states,” or rather those states that produce better results than Kansas when student demographics and population characteristics are kept constant.


But how do you do that?


In previous posts, when talking about the ACT and SAT metrics used, they are described as “adjusted for percent participation”.  Research has shown that the percent of students in a state taking either the SAT or ACT has a huge influence on the state’s overall results on these exams.  Therefore we employed the following method for ranking states on the results of these exams “controlling for” percent participation:


  1. Run a regression with test outcomes as the dependent (predicted) variable and percent participation as the independent (predictor) value.
  2. Use the results to predict the test outcomes that each state should have based on their percent participation.
  3. Compare each state’s actual outcomes with the predicted outcomes.
  4. Rank the states according to the difference between their actual and predicted outcomes.


Does anyone remember talking about line slopes and intercepts when you learned how to plot lines on a graph?  How about y = mx + b?  Anyone remember that?  


Well, essentially that is what we are talking about here.  The regression equation gives you the slope and intercept of a line, so then we can look and see on a graph where each state’s point should be, and then compare this to where their point actually is.   This difference; the space between the point on the graph and the line where we expected the point to be, can be thought of as the influence of all the other variables aside from the one(s) we are controlling for, as demonstrated in the following diagram:


 Scatter for Blog 2.JPG


To control for multiple variables at the same time, I used a multiple regression, which essentially does the same thing as described above but cannot be plotted on a 2 dimensional graph because there are multiple independent variables.  The steps involve:


  1. Putting all measures on a consistent scale (using Z scores),
  2. Running multiple regressions to determine the intercept and slopes for each independent variable included in the regression,
  3. Calculating the expected value of the dependent variable based on the values of the independent variables for each state and the intercept and slopes identified in #2,
  4. Determining the difference between the predicted and actual values for each state, and then
  5. Ranking the results.  


Confused?  Don’t feel bad.  It is confusing.  Let’s just summarize by saying that we calculated state rankings on student attainment and achievement measures controlling for student demographics and population characteristics.  


So, what do the Kansas rankings look like; both before and after controlling for these things?   



Outcome Measure
Unadjusted Rank
Rank Controlling for:
Student Demographics
Population Characteristics
Student Demographics and Population Characteristics
AFGR - All Students Rank
5
4
11
5
ACGR - All Students Rank
13
11
15
13
ACGR - Economically Disadvantaged Students Rank
13
13
14
12
ACGR - Limited English Proficiency Students Rank
5
7
3
3
ACGR - Students with Disabilities Rank
3
6
5
7
Percent of 18- to 24-year-olds who were HS completers Rank
28
30
26
27
NAEP Combined All Students Pct Basic Rank
13
6
10
7
NAEP Combined All Students Pct Proficient Rank
15
13
16
13
NAEP Combined NSLP Eligible Pct Basic Rank
12
12
12
11
NAEP Combined NSLP Eligible Pct Proficient Rank
13
16
17
17
NAEP Combined NSLP Ineligible Pct Basic Rank
6
3
5
2
NAEP Combined NSLP Ineligible Pct Proficient Rank
13
7
10
7
ACT Percent Meeting All 4 Benchmarks Adjusted Rank
11
10
13
9
SAT Mean Score - Combined Adjusted Rank
17
38
43
38


As you can see, overall the ranks actually don’t change a great deal.  When controlling for student demographics and population characteristics, Kansas’s ranks go up on 9 measures, stay the same on 2, and go down on 3 measures.  This means that Kansas has better student outcomes overall when controlling for student demographic and population characteristics than when these factors are not controlled for.


Going back to the method we used to identify Aspiration States and applying it to these two new sets of ranks, we get the following:  


Higher Impact States (better than KS on at least 8 out of 14 outcome measures controlling for student demographics and population characteristics)
  • Texas (11)
    • Spends more on 0/6 funding measures
    • Has more students per district, school, and staff
    • Better on 2/14 outcome measures without controlling
  • Kentucky (9)
    • Spends more on 0/6 funding measures
    • Has more students per district, school, and staff
    • Better on 3/14 outcome measures without controlling
  • Arkansas (8)
    • Spends more on 0/6 funding measures
    • Has more students per district and school but fewer students per staff
    • Better on 3/14 outcome measures without controlling
  • Maryland (8)
    • pends more on 6/6 funding measures
    • Has more students per district, school, and staff
    • Better on 3/14 outcome measures without controlling



TX
KY
AR
MD
Freshman Grad Rate
Cohort Graduate Rate
CGR - FRL
CGR - ELL
CGR - IDEA
Pct 18-25 - High School
NAEP Pct Basic
NAEP Pct Proficient
NAEP Pct Basic NSLP Eligible
NAEP Pct Proficient NSLP Eligible
NAEP Pct Basic NSLP Ineligible
NAEP Pct Proficient NSLP Ineligible
Pct Meeting All 4 ACT Benchmarks Adjusted
SAT Mean Score Adjusted


↑ = Value Higher than Kansas
↓ = Value Lower than Kansas
What does all of this mean?


First, when you control for student demographics and population factors, only four states outperform Kansas on a majority of student outcome measures.  Of these, Maryland spends more per pupil, while Texas, Kentucky, and Arkansas spend less.  Also of these, all have more students per district and per school, and all but one (Arkansas) has more students per staff.  


Second, none of the high impact states were identified as peer states, and none of the states originally identified as aspiration states show up as outperforming Kansas when you control for student demographics and population factors.  


Third, when looking at actual outcome values, none of the higher impact states outperform Kansas on more than 3/14 outcome measures.  This is an important factor to remember when comparing ourselves to these states.


Scholars, politicians, statisticians, advocates, educators, and other interested parties have been arguing about the connection between school funding and student outcomes for a long, long time, and it is unlikely that the debate will end anytime soon.  One argument often cited is that Kansas outperforms other states because our student population is “easier to educate.”  However, when controlling for those variables that supposedly make Kansas students more successful, Kansas still proves to outperform approximately 90% of the other states.

No comments:

Post a Comment