On the limits of LLM surprisal as a functional explanation of the N400 and P600

Abstract

Expectations about upcoming words play a central role in language comprehension, with expected words being processed more easily than less expected ones. Surprisal theory formalizes this relationship by positing that cognitive effort is proportional to a word’s negative log-probability in context, as determined by distributional, linguistic, and world knowledge constraints. The emergence of large language models (LLMs) demonstrating the capacity to compute richly contextualized surprisal estimates, has motivated their consideration as models of comprehension. We assess here the relationship of LLM surprisal with two key neural correlates of comprehension – the N400 and the P600 – which differ in sensitivity to semantic association and contextual expectancy. While prior work has focused on the N400, we propose that the P600 may offer a better index of surprisal, as it is unaffected by association while still patterning continuously with expectancy. Using regression-based ERPs (rERPs), we examine data from three German factorial studies to evaluate the extent to which LLM surprisal can account for ERP differences. Our results show that LLM surprisal captures neither component consistently. We find that it is contaminated by simple association, particularly in smaller LLMs. As a result, LLM surprisal can partially account for association-driven N400 effects, but not for the full attenuation of N400 effects. Correspondingly, this property of LLMs compromises their ability to model the P600, which is sensitive to expectancy but not to association.