Automated Short Answer Grading (ASAG) is increasingly deployed in high-stakes educational contexts, yet the dominant evaluation paradigm focuses primarily on agreement with human raters, overlooking robustness as a complementary quality dimension. Robustness, defined as the consistency of grading behaviour under controlled, meaning-aware input transformations, bears directly on construct validity, the degree to which an assessment measures what it claims to measure, and conditions the trustworthiness of automated decisions in education.
We propose a perturbation-first evaluation framework that assesses robustness across three validity-theoretic dimensions: invariance to construct-irrelevant variation, sensitivity to meaning-altering changes, and resistance to adversarial gaming. The framework combines rule-based perturbation generators, two-gate quality validation, and self-referential metrics that compare the grader against its own output rather than against gold labels, isolating robustness from accuracy. A dual-protocol design contrasts within-question evaluation, the optimistic scenario typically reported in the literature, with cross-question evaluation, the realistic deployment scenario, capturing how trained graders degrade under distribution shift.
We apply the framework to a benchmark ASAG dataset of science short answers and compare a trained machine-learning baseline with zero-shot Large Language Model (LLM) graders evaluated at two prompting levels, with and without access to a reference answer. Three findings emerge. First, trained graders exhibit substantial robustness degradation under realistic cross-question evaluation, suggesting that standard within-question benchmarks overestimate operational reliability. Second, zero-shot LLM graders exhibit stronger robustness on invariance and sensitivity dimensions while remaining comparatively less vulnerable to adversarial manipulation, although residual sensitivity failures display a stable hierarchy across perturbation types. Finally, providing reference answers introduces a subtle trade-off, with improved consistency on meaning-preserving variations but increased susceptibility to keyword-based gaming, which is invisible to single-metric evaluation.
We argue that multi-dimensional robustness evaluation should complement accuracy-based metrics in ASAG systems intended for high-stakes educational use, and we release an open-source framework that operationalises the approach and is freely available to the research community.