Tuesday, August 8, 2017

KSDE Measures of Postsecondary Effectiveness

This week, KSDE released information on their five identified outcomes for measuring progress. This blog talks about Postsecondary Effectiveness.  

KSDE utilized National Student Clearinghouse data that tracks high school graduates six years after graduation to determine how many Kansas public school students are graduating and going on to postsecondary enrollment and/or completion. 

The first important thing to note about this new measure is that it is informative and not punitive. After years of No Child Left Behind and Adequate Yearly Progress, school districts got used to fearing statistics from KSDE, because low performance could have serious financial and accreditation impacts. These outcome measures are in no way attached to financial decisions about each district, nor are they included as part of the state’s accountability to the federal government, as shown in the States ESSA plan. They are designed to give districts and schools indicators of how well they are doing and where they might want to focus their improvement efforts. 

The second important thing to note is that the Postsecondary Effectiveness rate is not used to compare all districts using the same absolute targets or guidelines. As we will discuss below, it uses several factors outside of the control of the school districts to predict how well students will do from graduation and beyond, then compares the predicted performance of these students with their actual performance. 

The methods KSDE has used for the Postsecondary Effectiveness rate are very similar to those KASB has used for predicting state performance as part of our annual Report Card and other analyses, and I feel what they have put together is a very good indicator of student postsecondary success for schools and districts to use. 

KSDE is providing a one-page summary of Postsecondary Progress at the district level to the public, and at the school level for those with access to their authenticated system (which is mainly school district staff and KSDE employees). The summary looks like this:




There is a lot of information contained in this one page, but they center around three key statistics: 
  1. Graduation Rate: KSDE continues to use the four-year adjusted cohort graduation rate, which calculates the number of students graduating with a regular high-school diploma within four years divided by the total number of students who were eligible to graduate. This calculation takes into consideration students moving into and out of the school or district and other factors that could impact the "on-time" graduation rate.  
  2. Success Rate: The success rate is defined as the number of students enrolled in a postsecondary institution and/or who earned a postsecondary degree or certificate or an industry recognized certificate within two years of high school graduation divided by the number of students who graduated from high school. KSDE determined where students went after graduation because they purchased the rights to use the National Student Clearinghouse data for Kansas. The Clearinghouse data, though not perfect, is the best available data on students as they move from high school into the postsecondary system.  
  3. Effective Rate: The effective rate combines the Graduation and Success rates (by multiplying them together) to indicate overall effectiveness in graduating students and getting them into a postsecondary education program.  
This overall calculated measure of effectiveness is not groundbreaking by itself, but KSDE did two other things with the rate that make it a much more useful indicator of school and district performance in this area: 
  1. They created a five-year average. KSDE calculated the three measures for the most recent five years for which data was available, and then averaged them. This is important because it shows how a school or district has been doing in recent history instead of providing a single snapshot of performance. 
  2. They predicted district and school performance based on external factors and compared the prediction to actual performance. This second point is the real key to this data's effectiveness and utility to districts. We will talk more about this piece below. 
A couple of years ago when KASB was looking for ways to expand our annual Report Card comparing Kansas to other states, we came up with the concept of a high impact state, which we defined as a state that had better student outcomes than what would have been predicted based on a variety of demographic factors, such as student poverty, percent of students in special education and ELL programs, median household income, population density, and adult educational attainment.  

Research shows the strongest predictors of student success are often those outside of the control of the state education system. Tlevel the playing field,” KASB controlled for these external factors by creating a regression equation that would predict each states performance based on these factors, then seeing how each state's actual performance compared to their predicted performance.  Those states noticeably above the line were considered higher impact, and those noticeably below the line were considered low impact.  

What we found, and why we ultimately decided not to include this calculation in later iterations of the Report Card, was the only states that had actual values further above the line than Kansas did were those states at the bottom end of the performance scales. Because these states actual performance was so low, we did not feel it was useful to compare Kansas to them and seek to learn from them. (You can read more about that analysis here.) 

Perhaps a simpler example of this method of looking at performance is how KASB ranks ACT and SAT results for our Report Card. Research shows the percent of a state's population taking these tests has a significant impact on the overall state results. States with lower percents taking the test have higher performance than states where all or most students take the test. So KASB calculates a regression line based on the percent participation, then compares the predicted values from that line to each state's actual values, then ranks the difference. (You can see last years Report Card here, and should watch the news for this years Report Card, which will be released soon.)  

KSDE took a very similar approach in their use of the Postsecondary Effectiveness Rate. They calculated a predicted effectiveness rate for each district and school using the following measures as predictors (taken from ksde.org): 
  1. Cumulative poverty: The proportion of the students’ school years spent in poverty. A school year in which the student spent some time eligible for free lunch was weighted more heavily than a year when a student spent some time eligible for reduced-price lunch.  
  2. Student mobility: The number of times these students changed schools during a school year.  
  3. Chronic absenteeism: The count of student years in which students missed at least 10 days or more in at least one school. 
  4. Expulsion and suspension rates: Total district count of expulsion and suspension events from 2012 through 2015.  

Like KASB, KSDE chose the measures for their model based on research showing they were significant predictors of postsecondary success.  

KSDE took the resulting regression model and used a 95 percent confidence interval to give districts some wiggle room, resulting in a predicted range for each district. For example, USD 501 (where one of my children goes and from which the other graduated) has a Predicted Effectiveness Range of 27.8 percent - 32.4 percent. This represents a predicted value of 30.1 plus or minus 5 percent. USD 501s actual Effectiveness Rate averaged across five years is 25 percent, meaning they are slightly below where they were predicted to be based on the cumulative poverty, student mobility, chronic absenteeism, and expulsion and suspension rates for the district.  

You might be wondering, What are districts and schools supposed to do with these results”? Again, much like what KASB has done at the state level, districts can compare themselves to other districts with similar external factors, and seek out those districts that seem to perform better than themselves for ideas on how they can improve the educational outcomes for their students.  

As important as this new data is for districts improvement efforts, it is also important to note how hard KSDE has been working to produce this information. They have made a great deal of effort over the past several years to seek input from schools, districts, and communities to determine what is important to Kansans when it comes to public education, and have worked to incorporate those values into their development of outcome measures specifically to inform and guide districts rather than to single out or punish them.  

For more information on the Postsecondary Effectiveness Rate, check out KSDEPostsecondary Progress User's Guide.

No comments:

Post a Comment