Wednesday, December 23, 2015

What a Tangled Web We Weave

One of the key debates in education is whether there is a causal relationship between education funding and student outcomes.  As I've discussed in previous posts, this is something that you can't really "prove" using statistics.  In this post, I want to expand on that by illustrating the issue of multicollinearity.

Multicollinearity, according to Wikipedia, is "a phenomenon in which two or more predictor variables in a multiple regression model are highly correlated, meaning that one can be linearly predicted from the others with a substantial degree of accuracy... the issue of multicollinearity arises when there is an approximate linear relationship among two or more independent variables."

So, aside from the limitations in the statistics themselves, which are only able to say whether two variables tend to move together (correlation) or whether movement in one can predict movement in another (regression), our ability to identify patterns is further complicated when the things we are using to predict are highly related to one another.

The following table shows the list of independent (predictor) variables we are currently working with, followed by a count of how many other independent variables each one is correlated with (statistically significant at .05 or less based on available data), along with which how many dependent variables and variables overall it is correlated.


The variables in the table are sorted based on how many of the dependent variables (student outcomes) with which each one is correlated.  As you can see, the spending variables rank pretty highly out of those which are at least in part under the control of the education system (highlighted in yellow).  

This means that education spending tends to move with student outcomes more consistently than some things like the percent of children in poverty.  (To see more details on the actual correlation coefficients, click here.)  

However, you will see that these funding variables are also significantly correlated with many of the other independent (predictor) variables.  For example, the table shows that The Current Spending Per Pupil RPP (state-cost-of-living adjusted) variable is significantly correlated with 15 of the independent variables.  They are:
  • Current spending per pupil RPP (of course)
  • Period (Year)
  • ACT percent of graduates tested*
  • Current spending per pupil
  • Median household income
  • Percent of 25 year olds and older with at least a:
    • High school diploma
    • Bachelor's degree
    • Graduate degree or higher
  • Percent of children in poverty*
  • Percent of public school students who are white
  • Percent of Students 3-21 served under IDEA
  • Percent of students eligible for free or reduced-price lunch*
  • Percent of students in English language learner programs*
  • Percent of the population under the poverty level within the past 12 months*
  • Population per square mile
  • SAT percent of graduates tested
  • Spending on instruction
  • Spending on instruction RPP
  • Student district ratio*
  • Student school ratio*
  • Student staff ratio*
  • Total revenue per pupil
  • Total revenue per pupil RPP
* Indicates a negative correlation, where an increase in one variable is accompanied by a decrease in the other. 

In fact, there were only six independent variables with which Current spending per pupil RPP was not correlated, and these were census population distribution variables that were only available for one year.  This means that, even though we call them "independent" variables, they are anything but.  We have to look at them in the context of all the other factors that impact student outcomes.  

Nonetheless, note that there is very little on the list that we can impact directly.  The only variables that we can directly change are those related to school funding, and those related to district, school, and classroom size.  KASB has been asserting that more funding is related to better outcomes.  Others have suggested that too much money is being spent on administration, and that larger districts are the answer.  The correlations support the former theory, but actually show that larger districts are associated with lower student outcomes.  

The KASB Research Department will continue to examine this data and provide analysis and perspectives to inform the ongoing debate.  

Thursday, December 17, 2015

New KPI poll shows Kansans have more confidence in local government than in the state

The following post presents research or analyses from outside KASB and is presented for information purposes.  KASB neither endorses nor refutes the conclusions or recommendations contained herein.
A new poll released by KPI this month shows that 25% of respondents agreed with the statement “Kansas state government operates pretty efficiently and makes effective use of my tax dollars,” whereas 45% agreed with the statement “Local government operates pretty efficiently and makes effective use of my tax dollars.”  


The poll contained 15 questions, many of which were aimed to see how much knowledge the respondents had about school funding, asking such questions as “How much funding per pupil do you think Kansas school districts currently receive from ALL taxpayer sources per year, including State, Federal and Local taxpayers?” and “Over the last 5 years, how much do you think total per-pupil funding has changed?”


Other questions were targeted very specifically, such as two that began with “Hypothetically speaking, if total taxpayer support of Kansas public schools were more than $13,000 per-pupil and school districts had also used more than 350 million of state and local taxes to increase cash reserves, how much would you agree or disagree with this statement…”  


Based on the premise above, the two statements given were “Funding for other state agencies should be reduced in order to give more money to local school districts” and “I would be willing to pay higher taxes in order to give more money to local school districts.”  Interestingly, 48% agreed with the first statement and 41% agreed with the second; compared to 39% and 50% that disagreed with each.  


Still others were based on data or premises that could be called into question, such as “Spending on out-of-the-classroom expenses - administration, building operations, transport, and food service - varies among Kansas school districts, up to as much as $8,000 per pupil” and “If an individual school district wants to spend more than is necessary to provide the same function or service, or add extras like retiree health care…”  

For more specific results from the poll, visit SurveyUSA’s site here.  

Thursday, December 3, 2015

Time to Regroup

This past summer, KASB revealed a method for identifying states to use for comparisons.  These included Peer States, Aspiration States, and Higher Impact States.  Since then, more recent data has become available, and KASB has had time to see how well our original calculations and comparisons have worked.  As a result, we have a) updated our original calculations based on the new data and b) modified the calculations and groups themselves based on experience.

Higher Impact States

The first notable change is that (for the time being at least) we are no longer using the "Higher Impact" comparisons.  The theory behind these was to look at states that have better student outcomes than would be expected based on their student demographics and population characteristics.  The problems with this grouping were:
  1. It was very difficult to explain to most audiences,
  2. It was difficult to calculate, making ongoing updates fairly time-consuming, and
  3. The states identified were for the most part those that had much lower student outcomes than Kansas and therefore it did not seem beneficial to look to them for ideas.  

Aspiration States

The second change relates to the Aspiration States.  The calculations for this group did not change, but there were updates to the following outcomes data:
  1. The percent of 18-24 year olds with at least a high school diploma
  2. Average freshman graduation rate
  3. NAEP
  4. ACT
  5. SAT
Based on these updates, the list of Aspiration States went from this:
  • New Hampshire
  • New Jersey
  • Massachusetts
  • Vermont
  • Minnesota
To this:
  • New Hampshire
  • New Jersey
  • Massachusetts
  • Vermont
  • Indiana
  • Iowa
  • Nebraska
So, Minnesota fell off the list, and Indiana, Iowa, and Nebraska were added.  This means that overall Kansas went from having five states with better student outcomes on at least 8 of the 14 measures used to having seven states with better student outcomes on a majority of these measures.

Peer States

The peer states have changed both in terms of updated data and modified calculations.  Originally the list was based on those states that were within half a standard deviation +/- of Kansas' value on a majority of the 10 student demographic and population characteristic variables used.  Of these, the following data was updated:
  1. Percent of Children in Poverty
  2. Percent of students who are white
  3. Adults (25 and up) with at least a high school diploma, bachelor's degree, or graduate degree
In addition, we identified seven new variables to use:
  1. Population per square mile in 
    • urbanized areas (50,000 or more people), 
    • urban clusters (at least 2,500 and less than 50,000 people), and 
    • urban areas (urbanized areas + urban clusters)
  2. Percent of the population in
    • urbanized areas
    • urban clusters
    • urban areas
  3. Percent of the population below poverty in the past 12 months
Finally, based on the addition of new variables, KASB decide to split the Peer States group into four types of peers:
  1. Student Peers - states with values within +/- 1/2 standard deviation of Kansas on at least 3/5 student demographic variables:
    • Percent of children at 100% poverty
    • Percent of students eligible for free or reduced-price lunch (at-risk)
    • Percent of students served under IDEA (special education)
    • Percent of students participating in English Language Learners program (bilingual)
    • Percent of student who are white
  2. Adult Peers - states with values within +/- 1/2 standard deviation of Kansas on at least 3/5 adult  demographic variables:
    • Median household income
    • Percent of 25 year olds and older with at least a:
      • High school diploma
      • Bachelor's degree
      • Graduate degree
    • Percent of the population with income below the poverty level in the past 12 months
  3. Distribution Peers - states with values within +/- 1/2 standard deviation of Kansas on at least 4/7 population distribution variables:
    • Population per square mile
    • Population per square mile in urbanized areas, urban clusters, and urban areas
    • Percent of the population in urbanized areas, urban clusters, and urban areas
  4. Overall Peers - states with values within +/- 1/2 standard deviation of Kansas on at least 9/17 of the Student, Adult, and Distribution variables.
The original list of Peer States was:
  • Oregon
  • Washington
  • Michigan
  • Nebraska
  • Pennsylvania
  • Wisconsin
  • Illinois

Overall Peers

The new list of Overall Peers is very similar; but Illinois was replaced by five other states:
  • Oregon
  • Washington
  • Michigan
  • Nebraska
  • Pennsylvania
  • Wisconsin
  • Alaska
  • Idaho
  • Iowa
  • Missouri
  • South Dakota

Student Peers

The Student Peers, or states with student populations very similar to Kansas, are:
  • Illinois
  • Michigan
  • Missouri
  • Rhode Island
  • Washington
  • Arkansas
  • Oregon
  • Virginia 
  • Wisconsin

Adult Peers

The Adult Peers, or states with adult populations very similar to Kansas, are:
  • Alaska
  • Illinois
  • Iowa
  • Michigan
  • Missouri
  • Nebraska
  • Oregon
  • Pennsylvania
  • South Dakota
  • Utah
  • Washington
  • Vermont
  • Wisconsin

Distribution Peers

The Distribution Peers, or states with similar urban/rural population distributions to Kansas, are:
  • Alaska
  • Idaho
  • Indiana
  • Iowa
  • Minnesota
  • New Mexico
  • North Dakota
  • Oklahoma
  • Missouri
  • South Dakota
  • Wisconsin
These different peer groups should allow us to look at the question "what do states similar to Kansas do?" in different ways.
Below is a map KASB prepared showing the new state groups, along with the difference between what each spends (in terms of total revenue per pupil) and what Kansas spends on education to share with the legislature and other interested parties.  Starting next week you will be seeing new information from KASB organized around these new state groupings.  Stay tuned!  

Monday, November 23, 2015

Yes, Virginia, there is a Correlation.

There has been some contention recently about a fairly straightforward question, namely “Is there a correlation between school funding and student outcomes?”

The answer is yes, and in this blog post, I am going to provide evidence to support this answer.

But before that, let’s make sure we all understand the question.  A correlation, according to Merriam-Webster, is “a relation existing between phenomena or things or between mathematical or statistical variables which tend to vary, be associated, or occur together in a way not expected on the basis of chance alone.”  

Therefore, to say two things are correlated is to say they tend to vary together.  When one is higher or lower, the other tends to be higher or lower. The correlation coefficient tells us the degree to which these things are correlated. A 100% correlation means that the two variables move perfectly in sync.  

Back to our question, what we are really asking is: “Do student outcomes tend to be higher/lower when funding is higher/lower?”  

Again, the answer is yes.  And I will show you by listing correlation coefficients for a variety of funding measures compared to a variety of outcome measures:

  • Total Revenue per Pupil
  • Current Spending per Pupil
  • Spending on Instruction per Pupil
  • Spending on Instruction as Percent of Current Spending
  • NAEP assessment results
  • ACT assessment results
  • SAT assessment results

This is all publicly available state-level data, and anyone can take a look for themselves.  The funding amounts come from the U.S. Census Bureau.  The adjustments for state cost of living were calculated using the Regional Price Parity statistics from the Bureau of Economic Analysis.  The NAEP data comes from the Institute for Education Sciences, and the ACT and SAT results come directly from the tests’ websites.  I used data across several years for all fifty states.  

Here are the results I got:

So, with the exception of the SAT, we can say that across all these student outcome measures, more money is correlated with better student outcomes.  We can also say that a higher percent of current spending going towards instruction is correlated with better student outcomes, but these correlations are not as high as those for the dollar amounts.

Does this mean increasing funding for education will lead to better student outcomes?  

No, it does not.  

The mantra that I have made everyone here at KASB who works with this data memorize is:  “Correlation is not causation.”  The fact that two things tend to move together does not mean one causes the other.  

With school funding and student outcomes, it is virtually impossible to prove via statistics that increasing funding will improve student outcomes.  But then again, it is virtually impossible to prove ANYTHING via statistics unless you are working in an environment where you have complete control over all your variables.  

School funding and student outcomes are also correlated with a lot of other things, like median household income, education level of adults, population per square mile, percent of students with disabilities, percent of students eligible for free and reduced-price lunch, percent of students eligible for ELL services, and so forth.  So to try and separate out the impact of funding on outcomes by itself is not something that can be done in a clear and concise manner.  Using different statistics for different time periods will yield different results, which is why people on both sides of the school funding argument continue to find new ways to present the data and argue their cases.

But let’s take a step back.  Do we really need statistics to tell us that if you increase funding to schools, schools are going to have a better chance of enabling students to succeed?  

You tell me.  

Wednesday, October 21, 2015

KIDS Count Data shows Kansas trends similar to Nation

“After numerous years of depressing economic news, many positive trends signal that the economy is finally recovering from the deep recession. Job growth and consumer spending are up, while unemployment is down. Nonetheless, there are warning signs that the recovery may be leaving the lowest-income families behind, disproportionately affecting workers of color and their children.”

So begins the narrative for the 2015 KIDS Count Data Book, produced by the Annie E. Casey Foundation.  The annual report gathers state-level statistics from many sources and produces overall ratings for each state and the nation. Some of the national trends highlighted in the report are listed below, followed by the information for Kansas.  

  • Worsening Nationally and in Kansas
    • The percent of children in poverty has increased from 18 percent in 2008 to 22 percent in 2013 nationally, and from 15 percent to 19 percent in Kansas.
    • The percent of children whose parents lack secure employment increased from 27 percent in 2008 to 31 percent in 2013 nationally, and from 22 percent to 24 percent in Kansas.
    • The percent of children not attending preschool increased from 53 percent in 2007-09 to 54 percent in 2011-13 nationally, and from 54 percent to 56 percent in Kansas.
    • The percent of children in single-parent families increased from 32 percent in 2008 to 35 percent in 2013 nationally, and from 28 percent to 30 percent in Kansas.
    • The percent of children living in high poverty areas increased from 11 percent in 2006-10 to 14 percent in 2009-13 nationally, and from 6 percent to 9 percent in Kansas.  
  • Improving Nationally and in Kansas
    • The percent of children living in households with a high housing cost burden decreased from 39 percent in 2008 to 36 percent in 2013 nationally, and from 28 percent to 27 percent in Kansas.
    • The percent of fourth graders not proficient in reading decreased from 68 percent in 2007 to 66 percent in 2013 nationally, and from 64 percent to 62 percent in Kansas.
    • The percent of high school students not graduating on time decreased from 25 percent in 2007-08 to 19 percent in 2011-12 nationally, and from 21 percent to 11 percent in Kansas.  
    • The number of teen births per 1,000 decreased from 40 in 2008 to 26 in 2013 nationally, and from 44 to 30 in Kansas.  
  • Differing Trends
    • The percent of teens not in school and not working was constant at 8 percent between 2008 and 2013 nationally, but increased from 5 percent to 6 percent in Kansas.
    • The percent of eighth graders not proficient in math decreased from 69 percent in 2007 to 66 percent in 2013 nationally, but remained constant at 60 percent in Kansas.
    • The percent of children in families where the household head lacks a high school diploma decreased from 16 percent in 2008 to 14 percent in 2013 nationally, but increased from 11 percent to 12 percent in Kansas.

Overall these results show Kansas follows national trends in most cases, but in the cases where Kansas differs from the national trend, Kansas is moving in a more negative direction. This data suggests more effort needs to be put into addressing the topics listed under differing trends above if the goal is to make sure Kansas follows national trends.  

For more information on Kansas’ data, be sure to check out the fact sheet prepared by Kansas Action for Children.

Wednesday, October 7, 2015

WalletHub ranks Kansas 9th for Teachers

The following post presents research or analyses from outside KASB and is presented for information purposes. KASB neither endorses nor refutes the conclusions or recommendations contained herein.
for your consideration.gif


WalletHub is a site for “information consumers and small business owners need to make better financial decisions and save money.” Some may recall that Governor Brownback’s statement that Kansas’ Education System ranked 5th in the nation was based on its annual rankings (the 2015 version of this report ranks Kansas at number 12).  


WalletHub recently released a report entitled 2015’s Best and Worst States for Teachers, in which they ranked Kansas 9th in the nation. The rank is a combination of two others:  a “Job Opportunity and Competition” rank of 23 and an “Academic and Work Environment” rank of 7.  


The Job Opportunity and Competition rank is based on the following measures, and was weighted twice as much as the Academic and Work Environment rank:


  • Average Starting Salary for Teachers (adjusted for cost of living)
  • Median Annual Salary for Teachers (adjusted for cost of living)
  • Teachers’ Income Growth Potential
  • Projected Number of Teachers per 1,000 Students by Year 2022
  • Unemployment Rate
  • 10-Year Change in Teacher Salaries (measures change in constant dollars for teacher salaries between the 2003–2004 and the 2013–2014 academic years)


The Academic and Work Environment rank was based on the following measures:


  • WalletHub “School Systems” Ranking
  • Pupil-to-Teacher Ratio
  • Safest Schools (percentage of public-school teachers who reported that they were threatened with injury by a student from school during the previous 12 months)
  • WalletHub “Underprivileged Children” Ranking
  • Public School Spending per Student (measures annual state and local expenditures for K-12 public schools per capita)
  • Average Commute Time
  • WalletHub “Working Moms” Ranking


The report does not provide each state’s ranks on these measures, but it does note that Kansas ranked 3rd lowest for pupil-to-teacher ratios.  


The report also included several experts’ answers to the following questions:


  1. What are the biggest issues teachers face today?


Responses include evaluation based on student performance, condition of school facilities, the need for better training, and restrictions based on standardized testing and policy.


  1. How can local officials attract and retain the best teachers?


Responses include improving work conditions, housing assistance, ample class prep time, supportive mentorship, and provide an active role in decision making.


  1. What tips can you offer young teachers looking for a place to settle?


Responses include considering district stability, ask other teachers, and avoid states pushing charters and vouchers.


  1. Are unions beneficial to teachers? What about to students?


Responses to this included positive things like protecting rights, advocating for fair compensation, and negative things like making it difficult to remove ineffective teachers and preventing school leaders from making productive changes.

For more information, check out the report here.  

Monday, September 28, 2015

The Race Achievement Gap

The following post presents research or analyses from outside KASB and is presented for information purposes.  KASB neither endorses nor refutes the conclusions or recommendations contained herein.
for your consideration.gif


The National Center for Education Statistics recently released a report entitled “School Composition and the Black-White Achievement Gap.” In it, they present results of analyses related to the impact of school composition on student outcomes. Put in plainer English, they looked at how the percent of White and Black students in a school impact student outcomes for Black and White students.  


Initially, the study found:


  • “Achievement for both Black and White students was lower in the highest Black student density schools than in the lowest density schools.”
  • “The achievement gap was not different [for black and white students].”


In other words, the higher the percent of Black students in a school, the lower the achievement scores were for both Black and White students. Why do you suppose that might be?  Here are some quotes from the article that might help explain:


  • “Schools that serve large percentages of Black students are more likely to employ less experienced teachers.”
  • “Schools with a higher percentage of students who are Black tend to have higher shares of low-socioeconomic-status students, who often need additional supports to be successful because Black students are more likely to be in a one-parent/guardian family, to be in a family in poverty, and to have parents with lower levels of education, compared with the parents of White students.”
  • “There is a body of work that explores whether negative educational outcomes, such as lower achievement, that are associated with large concentrations of Black students in schools might be due to an “oppositional culture,” which is a part of contemporary Black culture. This line of research considers student peer effects associated with larger concentrations of Black students where it has been theorized that certain behaviors that are associated with higher achievement are shunned because they involve learning to cope with pressures such as ‘the burden of acting White’.”
  • “Some researchers have considered whether teachers may also have lower expectations for student performance in schools with a high population of Black students, sometimes explained as a “Pygmalion effect.” This research is grounded in the assumption that lower expectations by teachers for students from minority backgrounds may result in lower levels of engagement by both teachers and students, which ultimately may contribute to poorer academic performance. One study found that in predominantly Black elementary schools, Black and White students tend to score lower and eventually are placed on a lower track in high school, and this tracking can start in elementary school.”
  • “At the high school level, some research shows that the tracking of Black students tends to differ by the density of Black students in the school. One study found that that Black students are more likely to be in high-track courses (e.g., taking algebra in the eighth grade rather than the ninth grade) in predominantly Black schools than in lower density schools. Another study found that even when controlling for achievement, more racial-ethnic and socioeconomic diversity are related to more “de facto” tracking.”
  • “The number of school disciplinary reports increases as the percentage of students in a school who are Black increases, and Black students are more likely than White students to face school discipline or office referrals (Rocque and Paternoster 2011), which is relevant because higher rates of out-of-school suspension are related with lower achievement.”


Based on these items, it would seem that there are many factors that could cause the achievement gap between schools that are predominantly Black and those that are predominantly White.  


The study also found that:
  • “Black students are, on average, in schools that are 48 percent Black, whereas White students are, on average, in schools that are 9 percent Black.”
  • “Schools in the highest Black student density category are mostly located in the South, with very few in the West.”
  • “Schools in the highest Black student density category are mostly in cities, but this varies by region.”
  • “Schools with higher Black student density also have higher percentages of students with low socioeconomic status.”


So, Black students are more likely to go to predominantly Black schools than White students are, schools in the South are more likely to be predominantly Black, and schools in cities are more likely to be predominantly Black than rural schools.  


But more significantly, there is a correlation between the density of Black students and the overall socioeconomic status of the school. Because a lot of previous research has shown a correlation between poverty and student achievement, it is possible that the connection between Black student density and student achievement could be based on their mutual connection to poverty.  


Because of this, the researchers went on to control for socioeconomic status (via regression analysis), and found that “the previously observed relationship between Black student density and achievement disappeared for Whites but not for Blacks.” When they controlled for socioeconomic status, student, teacher, and school characteristics, “ the achievement gap was greater among schools with the highest Black student density than the schools with the lowest.” So the achievement gap between Black and White students is higher in schools with a higher percent of Black students, even when controlling for poverty and the characteristics of the students, teachers, and schools.  


The study goes on to discuss differences in gender and between-school versus within-school differences.  


What does this all mean?


  • First, it again illustrates how complex issues like achievement gaps are, and points out how many interrelated factors are at work.  
  • Second, it tells us that the achievement gap between white and black students is worse in schools with a higher percent of black students, even when you control for many of the interrelated factors such as poverty.


When discussing education policy, we tend to talk a lot about students “in general” and try to think about education divorced from issues like race and poverty.  But the reality is that education is not simple or transparent, and it is important to take all factors into consideration.  

Friday, September 25, 2015

Best Practices for Teacher Merit Pay Systems

The following post presents research or analyses from outside KASB and is presented for information purposes. KASB neither endorses nor refutes the conclusions or recommendations contained herein.

As teacher merit pay continues to receive media attention in Kansas, KASB is reviewing pertinent research on the topic.  


A report released in February by the Center for American Progress (CEP) discusses lessons learned by 10 school districts that implemented teacher merit pay systems.  CEP is “an independent nonpartisan policy institute that is dedicated to improving the lives of all Americans, through bold, progressive ideas, as well as strong leadership and concerted action.”


The report indicated that all districts considered the following elements in their merit pay systems:
  • Base salary
  • Teacher effectiveness
  • Speed of salary growth
  • Career pathway opportunities
  • Incentives for hard-to-staff schools and positions
  • Bonuses, rewards, and recognition
  • Opt-in timeframe


From these components, the following best practices emerged:
  1. Differentiate compensation based on roles and responsibilities.
  2. Set starting salaries to meet market demand.
  3. Align teacher compensation redesign with fair and proven teacher evaluation systems.
  4. Shift pay away from years of experience and advanced degree attainment.
  5. Use compensation incentives to attract highly effective teachers to hard-to-staff schools, districts, and subjects.
  6. Emphasize extra pay for effectiveness and career pathways instead of small bonuses.
  7. Accelerate the timeline to earning the maximum salary where possible.
  8. Allow teachers to opt-in to new compensation systems within a set timeframe.

For more details, see the full report.