Thanks so much for the fun conversation @wiair.bsky.social ! A pleasure to chat about language, LLMs, and memory--covering some work with @bkhmsi.bsky.social @mschrimpf.bsky.social @michael-lepori.bsky.social @klemenkotar.bsky.social @evfedorenko.bsky.social @thomashikaru.bsky.social, among others!

Go, @andreadevarda.bsky.social! A beautiful and comprehensive study! 🔑 findng: behav. measures are ~fully reducible to simple predictors of processing effort (surprisal, word length+frequency), but for 🧠 measures, LLM embeddings carry additional predictive power, likely capturing aspects of meaning.

Wonderful work by @andreadevarda.bsky.social linking behavioral and neural manifestations of language comprehension using language models. With @rplevy.bsky.social and @evfedorenko.bsky.social

A similar asymmetry emerges within the effort measures. Eye movements are driven mostly by length and frequency (context-independent). Brain responses are driven mostly by surprisal (context-dependent). (9/10)

👀 Behavior: effort dominates. SFL (three numbers) performs better than 1600-dim embeddings, esp. in eye-tracking. Adding EMB to SFL gives only small gains. 🧠 Brain: the opposite. Embeddings predict fMRI/N400 responses far better than effort. (8/10)

We test this on 13 datasets: 👀 4 eye-tracking, 3 self-paced reading, 1 Maze, 🧠 1 ERP (N400), and 4 fMRI. All responses are averaged across participants and brain responses are averaged across voxels/electrodes so brain and behavior are comparable (1-dimensional). (7/10)

Our hypothesis: behavior shows a bottleneck. Rich representations get compressed into effort dimensions (SFL) before they can influence processing times. Brain responses have more direct access to the high-dimensional representations. (6/10)

We use LMs to operationalize effort and meaning in one framework. From the same GPT/GPT-2 models we get (i) surprisal (+freq and len; SFL) for effort, and (ii) contextual embeddings (EMB), high-dimensional vectors that encode form and meaning. (5/10)

But language is about transmitting meaning, and effort abstracts away from much of it. "The chef cooked the meal" and "The wolf caught the deer" have ~identical effort profiles but mean very different things. And brain studies show sensitivity to meaning besides effort. (4/10)

One influential view is that responses to language are driven by processing effort, mostly captured by three word-level predictors: surprisal, frequency, and length (SFL). These are the "Big Three" in reading research and they also predict brain responses. (3/10)

Psycholinguists study behavior (👀 eye movements, reading times). Neuroscientists study brain activity (🧠 fMRI, ERPs, etc.). Both make inferences about the human language system, but they are rarely studied together. Do they reflect the same information? (2/10)

New preprint! 🧠👀🤖 Behavioral and brain responses to language reflect different levels of linguistic representation w/ @whylikethis.bsky.social , @evfedorenko.bsky.social , and @rplevy.bsky.social (1/10)