Wednesday, June 17, 2015

What the Data Tells You, and What it Does Not

KASB has been working with the Kauffman Foundation on their EdWise initiative. As noted in this article on KASB's website, the purpose of EdWise is to provide “a more effective way to access public information in order to know more about the educational communities in Kansas and Missouri.” The folks at Kauffman have been very interested in learning more about the data and the actual phenomena behind it, and we see some great opportunities for collaboration with them.


The following is a visualization they created using Tableau, the same software that KASB is currently using to provide data to our members and other stakeholders. This one shows the college remediation rates as reported by KSDE:


https://public.tableau.com/views/PercentofKSHSGradsEnrolledinRemedialCourseworkinCollegebySchoolDistrict2007-2012/KansasMap


This tool accurately shows the remediation rates by district for 2007 through 2013. Further, the Kauffman Foundation provided the link back to the KSDE web page where the data came from, so anyone accessing this tool can determine the definition and know what the data is showing. You can read a blog post describing this tool in more detail here.


But what is this data not showing?


We all know that education is a very complex endeavor, and therefore the data collected and reported about that endeavor is likewise complex. The data shown above was collected as part of the requirements for the American Reinvestment and Recovery Act (ARRA) funding along with a host of other metrics the federal government defined in order to determine the effectiveness of the services being provided. To say that these measures were complex is like saying that the Titanic was a large-ish boat. I was part of the team at KSDE that worked to determine the best way to collect the data and report on these metrics, and we faced many challenges related to compliance with the federal definitions while making sure the information provided really painted an accurate picture of education in Kansas.


The statistic illustrated above is a very good example of this. Here are KSDE's definitions for the numerator and denominator of this remediation statistic:


Numerator: Number of students from denominator who enrolled in a Math, Reading, or English developmental course before or during their first academic year of attendance in a Kansas public postsecondary IHE following high school graduation.


Denominator: Number of students from the high school graduating class cohort entering postsecondary in the academic year following high school graduation and reported as degree or certificate seeking in postsecondary.


So, the percents are an indication of the portion of the graduating class that a) enrolled in college, and b) enrolled in a remedial course.


The first problem here is that we don't have information on the percent of high school graduates that enrolled in college in the first place. KSDE provides us the result of dividing the number of students who needed remediation by the number of students enrolling in college in the first place, but they don't give us these initial numbers for all years. Looking at KSDE’s site, this data is only available for 2012. To make matters more complicated, KSDE employs data suppression rules to protect student data privacy. These rules dictate that “data are displayed only when the number of students in a group is at least 10. Data for groups less than 10 are displayed as N/A.”  That means that you might have to dig in to your district’s local data to get this information.


Based on KSDE's data, we see that USD 348 Baldwin City and USD 385 Andover both have a remediation rate of 15 percent for 2012, which means that of their graduates enrolling in college, only 15 percent of them are also enrolling in remedial coursework. However, the additional data shows us that 69 percent of Baldwin City's graduates enrolled in college overall, and 88 percent of Andover’s graduates enrolled.


The other statistic available on the same page of KSDE’s site is the percent of graduates completing one year’s worth of college credit within two years. This data is only available for 2011, and KSDE uses the same data suppression rules on it.


Again using our example, 69 percent of USD 348's graduates and 74 percent of USD 385’s graduates complete their freshman year of college within two years of their high school graduation. So, looking at these additional pieces of information, we see that though the two districts share the same remediation rate for 2012, other statistics demonstrate differences in the percent of graduates enrolling in college and completing coursework. Plus, looking at the raw numbers we can see that Baldwin City had 100 graduates in 2012, and Andover had 371, so the size of the district and the graduating classes are very different.


It should also be noted that there is no statewide criteria for remediation. Each college or university determines how to identify students needing additional assistance as they enter school. There is likewise no standard number of credits that constitute a year's worth of college education. So the postsecondary institutions attended has a large impact on the remediation and completion rates. Finally, KSDE notes that the matching process for linking high school and college student information was imperfect:


“It is likely that the numbers and percentages of students enrolled in postsecondary and postsecondary remediation are underestimated in this report due to the match process. KSDE used both deterministic and probabilistic algorithms to match students with the KBOR dataset. However a student may provide a slightly different version of his or her name and demographic data to different educational agencies and institutions. This can result in the algorithms indicating less than a strong match and, as a result, the student would not be indicated as enrolled in a postsecondary institution.”


In summary, the Kauffman Foundation is making Kansas Education data more readily available to the public, which is a good thing, but it also means that we, as members of the school districts and school boards of Kansas, have a responsibility to first make sure we understand the data that is out there, and second to talk to other interested stakeholders and make sure they understand the data and the context for that data.


Below is a visualization I created based on the same data the Kauffman Foundation used from KSDE’s site, along with the additional information on percent enrolling and percent completing one year of credit.

No comments:

Post a Comment