Similarity measures in automated essay scoring systems: A ten-year review

In automated essay scoring (AES) systems, similarity techniques are used to compute the score for student answers. Several methods to compute similarity have emerged over the years. However, only a few of them have been widely used in the AES domain. This work shows the findings of a ten-year review on similarity techniques applied in AES systems and discusses the efficiency and limitations of current methods. In the final review, thirty-four (34) articles published between 2010 and 2020 were included. The metrics used to evaluate the performance of the AES systems are also elaborated. The review was conducted using the Kitchenham method, whereby three research questions were formulated and a search strategy was developed. Research papers were chosen based on pre-defined inclusion and quality assessment criteria. This review has identified two types of similarity techniques used in AES systems. In addition, several methods were used to compute the score for student answers in the AES systems. The similarity computation in AES systems is dependent on several factors, hence many studies have combined multiple methods in a single system yielding good results. In addition, the review found that the quadratic weighted kappa (QWK) was most frequently used to evaluate AES systems. © 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.