In previous posts, we’ve talked about identifying Peer States and Aspiration States, and about comparing their demographics, funding, and outcomes to Kansas. In this post, we’re going to talk about how to compare states on certain measures while removing the influence of others.
You may have heard people talking about “controlling for variables” or “holding variables constant” when discussing research, statistics, and analysis. Essentially this means you are trying to remove the influence of these variables on the relationship between other variables.
In the previous posts, we identified six categories into which our 44 statelevel measures fall:
 Student Attainment
 Student Achievement
 Money (School Spending)
 Student Demographics
 Organization Size
 Population
Further, we’ve had some discussions about how school spending relates to student attainment or student achievement.
However, often folks who oppose increases in school funding argue that a straight comparison of school funding to student outcomes ignores the fact that student demographics and population characteristics (among other measures) vary greatly from state to state, and further that differences in student outcomes are more likely due to these factors than to the amount of money spent.
In order to test out these assertions, we would need a way to remove the effects of student demographic and population measures on the student attainment and student achievement measures. Doing so will allow us to create a list of “higher impact states,” or rather those states that produce better results than Kansas when student demographics and population characteristics are kept constant.
But how do you do that?
In previous posts, when talking about the ACT and SAT metrics used, they are described as “adjusted for percent participation”. Research has shown that the percent of students in a state taking either the SAT or ACT has a huge influence on the state’s overall results on these exams. Therefore we employed the following method for ranking states on the results of these exams “controlling for” percent participation:
 Run a regression with test outcomes as the dependent (predicted) variable and percent participation as the independent (predictor) value.
 Use the results to predict the test outcomes that each state should have based on their percent participation.
 Compare each state’s actual outcomes with the predicted outcomes.
 Rank the states according to the difference between their actual and predicted outcomes.
Does anyone remember talking about line slopes and intercepts when you learned how to plot lines on a graph? How about y = mx + b? Anyone remember that?
Well, essentially that is what we are talking about here. The regression equation gives you the slope and intercept of a line, so then we can look and see on a graph where each state’s point should be, and then compare this to where their point actually is. This difference; the space between the point on the graph and the line where we expected the point to be, can be thought of as the influence of all the other variables aside from the one(s) we are controlling for, as demonstrated in the following diagram:
To control for multiple variables at the same time, I used a multiple regression, which essentially does the same thing as described above but cannot be plotted on a 2 dimensional graph because there are multiple independent variables. The steps involve:
 Putting all measures on a consistent scale (using Z scores),
 Running multiple regressions to determine the intercept and slopes for each independent variable included in the regression,
 Calculating the expected value of the dependent variable based on the values of the independent variables for each state and the intercept and slopes identified in #2,
 Determining the difference between the predicted and actual values for each state, and then
 Ranking the results.
Confused? Don’t feel bad. It is confusing. Let’s just summarize by saying that we calculated state rankings on student attainment and achievement measures controlling for student demographics and population characteristics.
So, what do the Kansas rankings look like; both before and after controlling for these things?
Outcome Measure

Unadjusted Rank

Rank Controlling for:
 
Student Demographics

Population Characteristics

Student Demographics and Population Characteristics
 
AFGR  All Students Rank

5

4

11

5

ACGR  All Students Rank

13

11

15

13

ACGR  Economically Disadvantaged Students Rank

13

13

14

12

ACGR  Limited English Proficiency Students Rank

5

7

3

3

ACGR  Students with Disabilities Rank

3

6

5

7

Percent of 18 to 24yearolds who were HS completers Rank

28

30

26

27

NAEP Combined All Students Pct Basic Rank

13

6

10

7

NAEP Combined All Students Pct Proficient Rank

15

13

16

13

NAEP Combined NSLP Eligible Pct Basic Rank

12

12

12

11

NAEP Combined NSLP Eligible Pct Proficient Rank

13

16

17

17

NAEP Combined NSLP Ineligible Pct Basic Rank

6

3

5

2

NAEP Combined NSLP Ineligible Pct Proficient Rank

13

7

10

7

ACT Percent Meeting All 4 Benchmarks Adjusted Rank

11

10

13

9

SAT Mean Score  Combined Adjusted Rank

17

38

43

38

As you can see, overall the ranks actually don’t change a great deal. When controlling for student demographics and population characteristics, Kansas’s ranks go up on 9 measures, stay the same on 2, and go down on 3 measures. This means that Kansas has better student outcomes overall when controlling for student demographic and population characteristics than when these factors are not controlled for.
Going back to the method we used to identify Aspiration States and applying it to these two new sets of ranks, we get the following:
Higher Impact States (better than KS on at least 8 out of 14 outcome measures controlling for student demographics and population characteristics)
 Texas (11)
 Spends more on 0/6 funding measures
 Has more students per district, school, and staff
 Better on 2/14 outcome measures without controlling
 Kentucky (9)
 Spends more on 0/6 funding measures
 Has more students per district, school, and staff
 Better on 3/14 outcome measures without controlling
 Arkansas (8)
 Spends more on 0/6 funding measures
 Has more students per district and school but fewer students per staff
 Better on 3/14 outcome measures without controlling
 Maryland (8)
 pends more on 6/6 funding measures
 Has more students per district, school, and staff
 Better on 3/14 outcome measures without controlling
TX

KY

AR

MD
 
Freshman Grad Rate

↑

↓

↓

↓

Cohort Graduate Rate

↑

↑

↑

↑

CGR  FRL

↑

↑

↑

↑

CGR  ELL

↑

↓

↑

↓

CGR  IDEA

↑

↓

↑

↓

Pct 1825  High School

↓

↑

↑

↑

NAEP Pct Basic

↑

↑

↓

↑

NAEP Pct Proficient

↑

↑

↑

↑

NAEP Pct Basic NSLP Eligible

↑

↑

↓

↑

NAEP Pct Proficient NSLP Eligible

↑

↑

↓

↑

NAEP Pct Basic NSLP Ineligible

↓

↓

↓

↓

NAEP Pct Proficient NSLP Ineligible

↑

↑

↓

↓

Pct Meeting All 4 ACT Benchmarks Adjusted

↓

↓

↑

↓

SAT Mean Score Adjusted

↑

↑

↑

↑

↑ = Value Higher than Kansas
↓ = Value Lower than Kansas
What does all of this mean?
First, when you control for student demographics and population factors, only four states outperform Kansas on a majority of student outcome measures. Of these, Maryland spends more per pupil, while Texas, Kentucky, and Arkansas spend less. Also of these, all have more students per district and per school, and all but one (Arkansas) has more students per staff.
Second, none of the high impact states were identified as peer states, and none of the states originally identified as aspiration states show up as outperforming Kansas when you control for student demographics and population factors.
Third, when looking at actual outcome values, none of the higher impact states outperform Kansas on more than 3/14 outcome measures. This is an important factor to remember when comparing ourselves to these states.
Scholars, politicians, statisticians, advocates, educators, and other interested parties have been arguing about the connection between school funding and student outcomes for a long, long time, and it is unlikely that the debate will end anytime soon. One argument often cited is that Kansas outperforms other states because our student population is “easier to educate.” However, when controlling for those variables that supposedly make Kansas students more successful, Kansas still proves to outperform approximately 90% of the other states.
 Higher Impact States Data Packet: https://www.kasb.org/assets/Publications/Research/StateComparisonHigherImpact.pdf
No comments:
Post a Comment