Tuesday, May 22, 2018

The Devil's in the Demographics

This scatterplot shows the relationship between the percent eligible for Free or Reduced-Price Lunch within a school and the percent performing at Grade Level or Above on the 2017 KS State Assessments (Reading and Mathematics). 

We all want to be able to compare things.  Humans have a drive to qualify the world around them by determining what is better and what is worse.  The problem is, we often make these judgments without considering all the pertinent facts.

Take school performance, for example.  What if you wanted to make a Report Card that compared schools on state assessment results?  What would you need to take into consideration?

From a statistical standpoint, you would want to find all the things that can reliably predict student outcomes on these state assessments not related to school effectiveness, then you would want to somehow take these into consideration when comparing one school to another, to make sure you were really getting a sense of the differences based on the school's influence rather than on other outside factors.

KASB has spent quite a bit of time with the Kansas State Assessment data for the past few years.  We've compiled in an online tool which can be found here:  https://public.tableau.com/shared/F7D4J2FZS 

In the process of examining this data, we have found several demographic characteristics at both the school and district level that are significant predictors of school performance.  In other words, we can reliably predict student outcomes on the Kansas State Assessments based on a variety of factors not directly related to school effectiveness.

Here are some examples of what we found:
  • Larger schools have lower percents of students performing at grade level.
  • Schools in districts with higher percents of white students have higher percents of students performing at grade level and at college and career ready. 
  • Schools in districts with higher percents of Hispanic students have lower percents of students performing at grade level and at college and career ready. 
  • Schools in districts with higher percents of ELL students have lower percents of students performing at grade level and at college and career ready.
  • Schools in districts with higher percents of Migrant students have lower percents of students performing at grade level and at college and career ready.
  • Schools with higher percents of Economically Disadvantaged students, which is measured by the percent of students eligible for Free or Reduced-Price Lunch, have lower percents of students performing at grade level and at college and career ready. 
  • Schools in districts with higher percents of Students with Disabilities have lower percents of students performing at grade level and at college and career ready. 

If all of the above are true, then a rating system that did not take these factors into consideration would be biased towards smaller schools with high percents of white students and low percents of students that are Hispanic, English Language Learners, Migrant, Economically Disadvantaged, and that have Disabilities.  

This is certainly something to consider when looking at any kind of rating system that claims to present an accurate comparison across schools in Kansas.  


Thursday, May 3, 2018

Kansas Enrollment Projections 2017-18 Enhanced

Earlier this year, KASB released its annual Statewide Enrollment Projections along with district-level projections by grade and free/reduced-price lunch status.  The report can be found here, and the online tool allowing you to select one or more districts can be found here.

Recently, KASB gathered demographics data from KSDE's report card data, and has incorporated this into the enrollment projections to provide state and district-level projections for gender, race/ethnicity, economically disadvantaged students, English Language Learner (ELL) students, migrant students, and students with disabilities.

This data was available for the 2012-13, 2013-14, 2014-15, 2015-16, and 2016-17 school years.  KASB calculated the percent change from one year to the next for these five years, then averaged them together to determine the 2017-18 values.  For 2018-19 through 2022-23, a five-year rolling average percent change was used to calculated the values.

It is important to note that these demographic indicators come from the state's assessment and accountability data, and may reflect a slightly different population than the headcount enrollment numbers, based on funding, that are used for the overall enrollment projection counts.

The results at the state level are as follows:


Gender


The data suggests there will be no change in the percent breakouts for gender, with just slightly over half of students (51.5%) male.

Race/Ethnicity


There is a growing percent of the student population reported as Hispanic, increasing from 17.8% in 2012-13 to 19.5% in 2016-17.  This trend is expected to continue, with the projections indicating the percent will increase to 21.8% by 2022-23.  Students reported as white or black are expected to decrease, but students reported in the remaining race categories, taken collectively, are expected to increase.

Economically Disadvantaged


From 2012-13 to 2016-17, the percent of Kansas students reported as economically disadvantaged decreased from 50.0% to 48.3%.  These projections suggest that by 2022-23, the percent could be down to 45.1%.

ELL


The ELL data shows the biggest change across this time period.  From 2012-13 to 2016-17, the percent of students receiving English Language Learner services increased from 8.4% to 11.8%.  If this trend continues, 17.3% of students could be receiving such services by 2022-23.  

However, it is important to note that the reporting and testing requirements have changed in recent years, as have the effectiveness of identifying students needing such services.  So, increases in the past few years may not continue throughout the projection period.  

Migrant


The percent of students identified as migrant continues to be very small - going from 1.8% to 1.9% in the actual data and projected to increase to 2.1% by 2022-23.  

Students with Disabilities


The percent of students with disabilities increased slightly in the actual data, and is projected to continue this gradual increase, going from 14.4% in 2016-17 to 15.5% in 2022-23.

Next year's projections will include these demographic categories, along with any others that KASB can find to include.  KASB also hopes to address the issue of virtual school enrollment numbers, which are reported by KSDE at the district level and can inflate projections for certain districts.


Friday, April 27, 2018

NAEP Limitations and Cautions

Every other year when the NAEP results come out, there is a lot of discussion and debate about the national examination.  Many put a lot of stock in the data it provides, as it is pretty much the only national test with results from every state that can be used to make comparisons of student performance across the country.  However, there are some limitations to the NAEP exam and some cautions that should be noted in terms of how the results are used.

In 2009, researchers from the Buros Center for Testing at the University of Nebraska-Lincoln (my graduate school) and from the Center for Educational Assessment at the University of Massachusetts - Amherst published the Final Report for their Evaluation of the National Assessment of Educational Process based on a congressional mandate.  The full report can be found here.

In the executive summary, the authors state the following:
Comparing student achievement on NAEP across states is complicated. To appreciate the challenges in making state-by-state comparisons, it is necessary to understand the sampling design adopted by NAEP and its potential impact on the results and their interpretations. In NAEP’s multistage cluster sampling procedure, not all students take the assessment, and those students who do take NAEP respond to a subset of the NAEP items in each content area. While this allows for a broad sampling of items from any one content domain, the extent to which subgroups of students are represented adequately in NAEP’s state samples is of concern.  
As reported in the current evaluation, NAEP’s sampling procedures do not ensure adequate representation of various subgroups (including those defined by race and ethnicity) within some states, putting valid interpretations about subgroup performances within a state and across states at risk. Using NAEP to verify state results regarding the achievement of students with disabilities is also problematic because decisions about inclusion and allowable accommodations are made at the state level. Because states vary in their inclusion rates and in their treatment of accommodations for NAEP, the validity of state-by-state comparisons is debatable. 
Below are the main concerns and recommendations the researchers stated regarding NAEP and appropriate uses of its data.

  • There is not an organized validity framework for the exam, which is needed given the complexity and multiple uses of NAEP.  According to the report, "An organized validity framework takes into account the history of the assessment program, current learning theory, and content-performance expectations from the subject-matter field and related professions. It also addresses contemporary xviii issues in current interpretations and uses of the assessment and anticipates future appropriate and inappropriate uses and consequences of the assessment."
  • Additional studies are warranted if NAEP is to be used to verify state assessment results.  According to the report, "As reported in the current evaluation, there are numerous factors that can jeopardize the validity of interpretations when using NAEP to verify state results. These include differences in content being assessed, differences in standard setting policies and procedures, differences in the definition of the achievement levels, and differences in the representation of the NAEP state samples. Additional alignment studies that evaluate the congruency between the content assessed by NAEP and state content standards and assessment are crucial. The sampling procedures for NAEP should also be studied. Representation of subgroups across states varies considerably as do the inclusion and exclusion rates for students with disabilities, impacting the validity of the use of NAEP results for state-by-state comparisons and for verifying state assessment results."
  • Revise review processes for NAEP technical reports and manuals that facilitate their timely release.   "Currently, release of NAEP technical documentation can be years after results have been released, exceeding what testing programs should tolerate...  There are several reasons for releasing timely technical documentation; primarily, it assists users in understanding appropriate uses and limitations of NAEP scores."
  • Other measures of U.S. students’ educational achievement do not provide strong sources of external validity evidence for NAEP achievement levels. "It is a challenge to gather validity evidence from multiple sources outside a standard-setting study that can be used to evaluate achievement levels. Furthermore, external data are not perfect evaluation evidence due to potential differences in content, sample, and purpose. For example, some tests (like well-known college admissions tests—e.g., the SAT and ACT) involve self-selected samples of college-bound seniors, not a nationally representative sample. In many cases external tests serve purposes that are very different from NAEP. As the differences between what tests purport to do and what they measure increase, the utility of these measures as external evidence decreases." 
  • NAEP should continue to explore methodologies for setting achievement levels. "Stakeholders continue to use achievement levels as one means of interpreting NAEP results. NAEP has engaged in extensive research on standard-setting since 1992 to improve its practice. Some of this research includes the pilot studies done on the new Mapmark method (Schulz 22 and Mitzel, 2005). However, because this new methodology is not widely used, more research on whether it is appropriate for other NAEP subject areas is needed. Although we conclude that the new methodology worked well with the experts involved in the study on the 2005 grade 12 mathematics assessment, the degree to which the method will work with experts from other subject areas cannot be determined from this evaluation." 
  • NAEP should prioritize gathering external validity evidence that evaluates the intended uses and interpretations of its achievement levels. "The validity evidence collected by NAEP from internal and procedural sources suggest that the methodology was implemented as intended and that panelists had a positive experience with the process. However, the reasonableness of the results is a judgmental decision by policymakers who should consider additional sources of information. External validity evidence is an additional source of information to help policymakers make the final policy decisions about NAEP achievement levels. Such evidence may include results from additional standard-setting methods, state university entrance levels at the high school level, and transcript studies that evaluate course performance.12 The extent to which the sources of evidence may converge is affected by the intended uses and interpretations of NAEP’s achievement levels as articulated in a validity framework."
  • Current NAEP inclusion and participation policies and rates may not provide evidence to support intended uses and interpretations of NAEP. "As mentioned earlier, the intended uses and interpretations of NAEP results should be defined in a validity framework and related to how different types of students and schools are included in the results. Unlike state assessment programs developed for NCLB, all students do not take NAEP. Further, those who take NAEP do not take a full assessment but rather a sample of its content. Thus, those included or excluded can influence the results and any score interpretations. This is particularly true for students with disabilities (SWD) and English language learners (ELL). Decisions about inclusion and accommodations of SWD and ELL are made at the state level... Beyond inclusion policies, participation is also an important consideration. NAEP remains a voluntary assessment for students. Therefore, nonresponse and refusal to participate represent potential threats to the validity of NAEP scores, particularly for grade 12 and private school samples. For example, Chromy (2005) noted that recent student participation rates for grade 12 (74 percent) were considerably lower than grade 4 (94 percent) and grade 8 (92 percent). It is also unclear whether current sampling plans include all potential subgroups of interest within a state, such as students with specific ethnicities, disabilities, varying language proficiencies, and free and reduced-priced lunch program status."
  • Intended users were not familiar with NAEP scale scores and had difficulty distinguishing between achievement levels on NAEP and those that were developed by states for NCLB reporting purposes. "Most participants in our utility studies identified NAEP with state-level results. This represents a communications challenge for the future because of stakeholders’ familiarity with the reporting scales and achievement levels used for their state’s own NCLB assessment. For example, there was confusion among participants between state and NAEP achievement level results. This led to recognition that states’ definitions of Proficient are perhaps different from NAEP’s definition of Proficient. However, the nature of such differences is not readily apparent. Another source of confusion is that NAEP defines three achievement levels (i.e. basic, proficient, and advanced), yet often indirectly reports student performance at four levels (i.e. below basic, basic, proficient, and advanced). No policy definition for the achievement level below basic exists." 
  • Prioritize score reporting and interpretation as an area for research in the NAEP program. "Systematic studies of methods to report NAEP scale scores and achievement levels should be carried out with stakeholder groups prior to their operational use. Although some of this research may include print media, a more critical focus for evaluation is the expanding presence of NAEP on the World Wide Web. Where appropriate, the NAEP elements on the Web should be revised to represent empirical findings about ease of use, stakeholder interests, and accepted Web site development practices. Because NAEP reporting continues to invest in the use of interactive, online tools, the utility of these features must also be assessed." 
It is important to note that this report is almost ten years old, so it is possible that some of the concerns listed above have been addressed.  However, I could not find any research or reports that described changes like the ones suggested above in the years since it was published, and most sources indicate very little has changed about how NAEP is administered, scored, and reported in the past several years. 

Monday, April 23, 2018

Mind the Gap

Last time, we talked about the difference between Kansas and all Public Schools in the U.S. in terms of the NAEP reading and math assessments for 4th and 8th grade.  This analysis looked at the values for all students.

Today, we are going to look at the gap in performance between those who qualify for free or reduced-price lunches under the National School Lunch Program versus those who do not qualify.  For public school students in the nation, this is the closest thing we have to a direct measure of students' socioeconomic status.  The gap between performance for high and low income students indicates how successful the education system is at providing an equitable education for all students.

The comparison group we are using for this analysis is slightly different than what we used for the last one based on the available data.  Previously we were able to only include public school numbers at the national level, but for this analysis we will be looking at the total U.S. average for comparisons.

So, this analysis will examine the question, "Is Kansas doing better at providing equitable education for low and non-low income students than the nation as a whole?"

Scores

The chart above shows the average gap between scores for students eligible for free or reduced-price lunch and scores for students not eligible for free or reduced-price lunch.  The orange lines indicate the average for the United States, and the blue lines indicate the averages for Kansas.

The first thing to notice is that across time, the gap has been fairly consistent at around 20-25 points.  The second thing to notice is that over time, the gap for Kansas has been very close to, but just under, the gap for the United States.

From 2015 to 2017, however, the gap for Kansas students taking the 4th grade math exam decreased.  In 2015, the gap for Kansas was closer to the U.S. average than in any previous year, but it started moving away again in 2017.  This patterns was similar for 4th grade reading, but the gap for Kansas was actually higher than the gap for the U.S. in 2015, but dropped back below it in 2017.

The gaps for 8th grade reading and math declined for Kansas in 2015, but increased again in 2017 to almost the same as the U.S. average.

This data suggests that Kansas Public Schools have historically been about as successful at ensuring equitable education for low income students as the nation has, and that the gap in scores suggests more efforts are needed.  In addition, the gap for 4th grade students in Kansas is decreasing, while the gap for 8th grade students is getting bigger.

Percent at Basic or Above

The chart above shows the average gap between the percent of students eligible for free or reduced-price lunch who scored at Basic or above and the percent of students not eligible for free or reduced-price lunch who scored at Basic or above.  The orange lines indicate the average for the United States, and the blue lines indicate the averages for Kansas.

The patterns seen for these percentages closely follows those described for the scores; Kansas has historically had a smaller gap than the nation, but they have gotten closer together in recent years.  In 2017, 4th grade results showed a decrease in the gap for Kansas, but 8th grade results showed an increase in the gap for Kansas (after decreases seen from 2013 to 2015).

This data again suggests that the gaps in performance, both for Kansas and for the nation, are too large, and have shown little substantial change over time.  In Kansas,  this data suggests more efforts need to be made in the higher grades to see a decrease in the gap, and that the efforts being made in the lower grades needs to continue.

Percent at Proficient or Above

The chart above shows the average gap between the percent of students eligible for free or reduced-price lunch who scored at Proficient or above and the percent of students not eligible for free or reduced-price lunch who scored at Proficient or above.  The orange lines indicate the average for the United States, and the blue lines indicate the averages for Kansas.

Though the patterns seen for this metric are very similar to the trends described for the scores and for the percent performing at Basic or above, the differences between Kansas and the U.S. are much smaller for the students performing at Proficient or above.  Further, the gap for 4th grade reading was higher for Kansas than for the nation in 2013 and 2015, but dropped below the national average in 2017.

Summary & Conclusion

Taken all together, these comparisons suggest that:
  • Nationally, efforts are needed to help lower income students perform at the same levels as higher income students.  
  • The differences between how lower and higher income students perform in Kansas is very similar to the differences nationwide.  
  • Historically the differences between lower and higher income students' performance in Kansas has been lower than the differences nationwide, but recent years have seen Kansas' gap exceed that for the nation.
  • From 2015 to 2017, in general the gap for Kansas students in 4th grade decreased, and moved further below the national average than in recent years.
  • From 2015 to 2017, in general the gap for Kansas students in 8th grade increased, and moved closer to the national average than in recent years.
In Kansas, we have spent much of the last decade discussing the importance of an education that is both adequate and equitable.  The NAEP data suggests that, whether the education is adequate or not, it is not equitable.  The Kansas legislature and public school system have a responsibility to make efforts to improve equity, but it is important to note that the discrepancies we see in Kansas closely mirror those seen nationwide.  This suggests that national efforts are also needed to improve the education that lower income students are receiving to enable them to perform as well as their higher income peers.  

Tuesday, April 10, 2018

How Kansas Compares to the Nation’s Public Schools on NAEP


Introduction


The 2017 NAEP results were made available this week, so many are digging through the data to try and make sense of what it might tell us.  This article is another attempt to do just that.

KASB is frequently asked to speak to how Kansas students are doing.  It would make sense to answer this question by comparing Kansas public school students to all public school students in the nation, and to focus not on the scores and percentages themselves, but on how much above or below the nation Kansas students perform. 

Therefore, for this analysis, I am presenting the difference between the Mean Scores, Percent at Basic or Above, and Percent at Proficient and Above on the 4th and 8th grade Reading and Math exams for Kansas public school students and the scores and percentages for all public school student nationwide.

There are student subgroups worth investigating, such as students eligible for free or reduced-price lunch versus those who are not eligible.  Though this article focuses on all students, further analysis may delve into these subgroups.

Mean Scores


The following graph shows the difference in mean scores between Kansas and the nation’s public schools on the four exams.



As can be seen, in terms of mean scores, Kansas students have had consistently higher mean scores than the national average for public school students with the exception of the 4th grade reading test in 2015.  However, the number of points above the national average has been declining since 2005.  That is, until 2017, when the number of points above the national average increased for all tests except for the 8th grade reading exam, which is also the only exam that saw an increase in the number of points Kansas is above the nation in 2015.

This data suggests that though Kansas has been “losing it’s lead” on the nation for several years, this trend may be reversing as of the 2017 results in terms of mean (average) scores.

Percent at Basic or Above


The following graph shows the difference in the percent of students performing at Basic or above between Kansas and the nation’s public schools on the four exams.



Similar to the mean scores, this data shows that the percent of Kansas public school students performing at Basic or above has been consistently above the national percentage (except for 4th grade reading in 2015), but Kansas’ lead has been declining since at least 2009.

For 4th grade, the number of percentage points above the national percent increased from 2015 to 2017, suggesting a reversal in the trend.  For 8th grade, Kansas’ lead decreased slightly for Math, and more noticeably for Reading.

Percent at Proficient or Above


The following graph shows the difference in the percent of students performing at Proficient or above between Kansas and the nation’s public schools on the four exams.



As with the other two metrics, the percent of Kansas students performing at or above Proficient has been consistently higher than the percent of public school students nationally.  The number of percentage points above the national average has been generally on the decline since 2005.

However, for 4th and 8th grade math and 4th grade reading, the number of percentage points above the national average increased for Kansas public schools, and the 8th grade reading number declined only slightly. 

Conclusion


This brief analysis suggests that the last decade has seen a change in the performance of Kansas public school students when compared to public school students nationwide, with Kansas students continuing to outperform the nation, but losing their lead in each subsequent testing year.  However, the 2017 data suggest that there may be a change in this trend, with Kansas students starting to push ahead further than their peers in other states to a greater extent than in 2015. 

With new funding for Kansas public schools, we can only hope that this new trend continues.

Thursday, September 28, 2017

The Education Commission of the States - Recent Research

I sometimes use the Sticky Note application on my desktop computer.  Not to store all my passwords, of course, but to keep track of interesting articles or studies that I want to look into "when I have time." 

One of my favorite sources for such articles is the Education Commission of the States.  The commission was started in the mid-sixties as a way for scholars and policy makers to share information and ideas across the country.   Today they produce great research and analysis on education policy and education system characteristics in each of the 50 states. 

Below are five studies they have released recently that have been on my "to review more in-depth when I have time" pile.

High School Feedback ReportsEducation Commission of the States researched high school feedback reports in all states to provide this comprehensive resource. This database reviews high school feedback reports and systems in all 50 states. A state received a “yes” if the state has established a system or report that provides data regarding postsecondary enrollment and/or postsecondary performance of high school graduates, broken down at the high school and/or district level.

Key takeaways

  • Forty-two states currently have a system or report that provides data regarding postsecondary enrollment and/or postsecondary performance of high school graduates, broken down at the high school and/or district level.
  • Twenty-four states provide one or more elements of this data broken down by race/ethnicity.
  • Twenty-one states provide one or more elements of this data broken down by income indicators.
  • Thirty-nine states’ high school feedback reports are publicly available.
  • State education agencies produce high school feedback reports or systems in 21 states, state higher education agencies produce them in 13 states, and P20W collaborative agencies or statewide longitudinal data systems produce them in eight states.


Advancing Student Success Through the Arts: This Education Trends report explores research on how the arts bolster the development of deeper learning skills, provides examples of programs that successfully increased access to the arts in education in public schools, and includes state- and local-level policy considerations.

State Innovations for Near Completers: This Promising Practices report overviews the 2017 policy landscape regarding near-completers, reviews three states’ outreach strategies to this population and includes policy considerations for state leaders.

The Civics Education Initiative 2015-17:  This Education Trends report provides an update on state adoption of the Civics Education Initiative, explains the impact of the initiative, looks at how states customized the initiative, and provides examples and opportunities for policymakers to build on civic education policy efforts, such as the Civics Education Initiative.

Outcomes-Based Funding: This Policy Snapshot defines and explores outcomes-based funding, and provides summary information on 2016 and 2017 legislative activities.

If you visit ECS's site, be sure to subscribe so you will receive notices when they release new reports.

Wednesday, September 20, 2017

KASB Calendar Survey Data – Changes from 2016-17 to 2017-18

KASB recently released the 2017 Annual Calendar Survey Report, where we describe the information from districts related to the 2017-18 school year and all school years back to 2001-02.  In it, we discuss trends across this period, but do not focus on the changes from last year to this year. 
In this post, we will talk about what has apparently changed since last year, and why these changes might have happened.

Note that the data being used for the annual report and for this post represents 97% of the districts for 2016-17, and 77% of the districts for 2017-18, so some of the differences could be due to the districts who have not responded yet for this year.

Here are some of the changes from last year to this year that are worth noting:


  • Days:  The data shows that from 2001-02 through 2016-17, the average number of student contact days decreased steadily from 177.4 to 167.5, but then increased to 168.2 in 2017-18.  This is the first year the student contact days have increased since 2005-06.  In addition, in-service days increased from 6.4 last year to 6.7 this year, which is the highest average number of days reported so far on this survey.
  • Duration: The data shows that from 2001-02 through 2016-17, the number of hours spent at school per day by both students and teachers has been steadily increasing, with teachers spending about 10 minutes more per day at school in 2016-17 than they did in 2001-02, and with students spending about 15 minutes more.  However, from 2016-17 to 2017-18, the average time spent at school has decreased one minute for both students and teachers. 
  • Kindergarten Formats: The percent of kindergarten programs that are all day, every day has been increasing since 2001-02, with 34% of schools reported with full day programs in 2001-02 compared to 88% in 2016-17.  However, the percent of schools reported with all day every day programs increased to 94% in 2017-18, which is a much larger increase in percent than has been seen in recent years. 

In previous reports, we have noted that the general trend in Kansas was that teachers were working fewer days per year, but those days were longer.  One possible explanation for this is districts’ needs to save money by decreasing teacher contract days, and another is that districts decreased teacher contract days in lieu of salary increases as part of the teacher contract negotiations process. 

If either or both of these explanations are true, then the change in trends for days and duration could be due to the additional money districts received for the 2017-18 school year after several years of flat funding.  Districts may be moving towards having more student contact days and more teacher in-service days while shortening the actual school days themselves because they have the flexibility to do that with the additional funding received. 

In terms of Kindergarten formats, at the state and national levels, it has for some time been the consensus that all day, every day kindergarten formats are preferable to half day every day programs, all day every other day programs, or even half day every other day programs.  However, the state has not made the commitment to fully fund all day every day kindergarten programs until this year.  The larger increase in the number of schools reported with all day every day kindergarten programs is likely due to the additional money districts received for the 2017-18 school year for kindergarten classrooms.

To read the full KASB 2017 Calendar Survey Annual Report, click here.