“Psychometric Calibration of Large Language Models for Multidimensional L2 Academic Writing Assessment: The LICA Framework” (2026), by Pegah Merrikhi. Arts and Humanities research paper. Original abstract below, unchanged. A free PDF is available.

Psychometric Calibration of Large Language Models for Multidimensional L2 Academic Writing Assessment: The LICA Framework

Pegah Merrikhi · October 28, 2026 · Arts and Humanities

Abstract

The rapid integration of frontier Large Language Models (LLMs) into Automated Writing Evaluation (AWE) has raised pivotal questions regarding their psychometric validity, construct representation, and rater consistency in second language (L2) writing assessment. While LLMs offer scalable scoring and feedback capabilities, empirical evidence regarding their susceptibility to systematic scoring drifts across nuanced analytical criteria remains scarce. This calibration study benchmarks three frontier LLMs—GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro—against an expert human gold standard using the multidimensional LICA framework (Language, Ideas, Cohesion, and Academic Appropriateness). Utilizing a rigorous psychometric pipeline spanning ordinal alignment (Quadratic Weighted Kappa [QWK] and Mean Absolute Error [MAE]), inter-rater reliability (two-way random-effects "ICC" [2,1]), systematic scoring bias (Bland–Altman agreement limits), and multi-rater consensus (Fleiss’ κ), we evaluated model behaviors across a calibrated corpus of L2 academic essays (N total =20 rating instances per model). Empirical findings demonstrate marked dimension-dependent divergence in automated assessment: GPT-4o achieved deterministic concordance with expert ratings ("QWK"=1.000, "MAE"=0.000, "ICC"=1.000, and zero directional bias: d ̅=0.000), faithfully replicating human evaluative criteria. Claude 3.5 Sonnet displayed robust ordinal reliability ("QWK"=0.949, "ICC"=0.966), yet exhibited a systematic leniency drift localized to Cohesion and Academic Appropriateness (d ̅=+0.100, "MAE"=0.100). Gemini 1.5 Pro manifested a pronounced severity drift across Language, Ideas, and Appropriateness (d ̅=-0.250, "MAE"=0.250, "QWK"=0.925, "ICC"=0.947). Crucially, autonomous multi-model consensus reached substantial agreement on overt structural and conceptual dimensions (Ideas: κ=0.643; Cohesion: κ=0.643; Language: κ=0.545), but degraded considerably on Academic Appropriateness (κ=0.286, fair agreement). These results reveal a fundamental construct dissociation: while current generative LLMs reliably evaluate codified linguistic and rhetorical patterns, they struggle to consistently appraise sociopragmatic nuance, epistemic stance, and contextual register conventions. The study cautions TESOL practitioners and assessment developers against treating LLM ratings as monolithic constructs, highlighting the critical imperative of dimension-specific rater calibration and human-in-the-loop oversight in high-stakes L2 writing evaluation and formative automated feedback systems.

Free PDF DOI: 10.21203/rs.3.rs-11277915/v1