The benchmark derives controllable families of grade-school mathematics problems from symbolic templates. This lets evaluators vary numerical values, clause counts and distractors while preserving the underlying reasoning structure, testing whether model success is stable rather than tied to a…
Apple researchers introduced GSM-Symbolic and reported substantial performance variance across different instantiations of the same GSM8K-style problem templates, with further degradation as problem complexity increased and with irrelevant-but-plausible clauses causing drops of up to 65% in the evaluated models.
The benchmark derives controllable families of grade-school mathematics problems from symbolic templates. This lets evaluators vary numerical values, clause counts and distractors while preserving the underlying reasoning structure, testing whether model success is stable rather than tied to a fixed benchmark instance.
GSM-Symbolic made benchmark robustness itself a first-class question in AI4Math evaluation and provided a reusable way to distinguish headline accuracy from consistency under controlled perturbations.
Significance rationale: GSM-Symbolic shifted mathematical-reasoning evaluation from a single static score toward controlled robustness tests by regenerating semantically equivalent problems from symbolic templates and measuring sensitivity to changed values and irrelevant clauses.
—
The public research record introduced GSM-Symbolic and reported robustness results across controlled variants of GSM8K-style problems.
Apple's public repository exposes the benchmark templates and generated GSM-Symbolic, P1 and P2 data. E3 records inspectability and does not imply independent verification of every reported model result.
No corrections recorded.
