Rater Reliability in Special Education

  Throughout the COVID-19 pandemic, as a teacher, it was interesting to see the aspects of education that changed, and the aspects that did not. One thing that did not change that I continuously struggled with was the use of rater scales used by school psychologists to qualify students for social-emotional special education services.

In special education, when students are considered for social emotional services, a rater scale is to be filled out by parents and teachers, and then the student’s rating on this is compared to kids the student’s age. During the school year, this method is effective enough, but some of the measures had become very unreliable with the situation the COVID school closure put us in, and it leaves many parents and teachers in a pickle. 

At the end of the year, it was time for one of my kindergarteners to be reevaluated for his special education services. Though he had not previously qualified for social emotional services, he was on the autism spectrum, so the school psychologist asked for him to have a rating scale filled out. However, here is the kicker- neither me or his other teacher had seen him in person at all! 

As I go through the rater scale, which I have done for many of my students, it hits me how unreliable this measurement is for such a high stakes decision. Some of the questions are: does he hold a spoon, does he carry change? Does he know his address? Again, this is a kindergartener I have never met in person, but we have to rate him on a scale from never to always on things we have never seen.

Then, we get to his evaluation meeting.  His rating shows he can not do things (some of them many kindergarteners can not do). The parents are understandably confused why their ratings of their child is different from the teachers’. He also qualifies for social emotional services. Though I recognize that this is an unreliable measure, it is something my school psychologist uses, though I do not support it. 

It is important to understand aspects that can make rater scales horribly unreliable. For me, I recognize now that my rater error had to do with personal bias in which, “teachers ten to rate based on inappropriate or irrelevant stereotypes” (Brookhart & Nitko, 2019, p. 290).  For me, I knew the student was a kindergartener on the autism spectrum. I did not know many kindergarteners who know their address, for example, so I based my ratings on that. Additionally,  all three of us raters did not have consistent results, meaning that there was no inter-rater reliability. That should have been something communicated and discussed by us as his IEP team.

Perhaps I should have refused to rate, knowing that I have not seen this student in their natural setting, or had time to observe many questions asked on the rating scale. One thing is clear to me. We need to advocate for high stakes decisions to be made for students based on an unreliable measure, such as the mistake I made. In recognizing the unreliability, I have the power to speak up and advocate for my students. A power I will use in the future.  

Brookhart, S. M., & Nitko, A. J. (2019). Educational Assessment of Students (8th Edition) (8th ed.). Pearson.