Many-Facet Rasch Models in the Analysis of High-Stakes Tests

Constructed-response and performance-based assessments are commonly used in many educational measurement contexts. Because the scoring of these assessments is subjective and typically involves human raters, the results are often subject to scrutiny due to rater-related errors. For this reason, rater analysis and the examination of rater scores have become important considerations in various measurement theories. In this study, data from the Animation practical exam, administered by the National Organization for Educational Testing (NOET) to evaluate applicants for higher education programs in Animation, were analyzed using Many Facets Rasch models. The results indicated that rater severity was moderate, ranging from -0.02 to 0.05, suggesting the absence of systematic leniency or severity biases among the raters. However, rater consistency analysis revealed evidence of range restriction error. The test items were found to be relatively easy, with satisfactory levels of item discrimination. Moreover, the generalized Many Facets Rasch model (GMFRM) demonstrated significantly higher accuracy in estimating ability parameters compared to the traditional Many Facets Rasch model (MFRM). Furthermore, model fit indices (Akaike Information Criterion [AIC] and Bayesian Information Criterion [BIC]) indicated a superior fit for the GMFRM compared to the MFRM.