Agreed they're a nice controlled case to study!

But I think that's a "the science is far from complete because we just started studying these systems" issue, not an "it's impossible to do science on generalization in these systems" issue. 4/4

Now, I do think for things like robustness or mathematical creativity you *do* need sufficient training diversity, which as you say requires frontier models, and makes it harder to make as concrete of links between them and smaller experiments just yet. (Obv. lab secrecy makes it harder.) 3/4

I agree with most of what you said, and especially that these are good questions to ask! I guess a core premise of (part of) my research program has been that we *can* study these issues by things like creating controlled, novel tests on which to evaluate generalization of large models 1/4

Indeed, I guess everyone probably has the ick response at some form of reference to AI. But it doesn't seem ideal for our discussions to revolve around people's emotional reactions (though I'm also not sure how to avoid it)

I do think that people have an ability to continue improving in an area of their expertise in a way that current models cannot as effectively, but I'd argue what they do is closer to moving the domain in distribution by working on it than generalizing OOD zero-shot.... 4/4

And similarly the people who prove something like the kinds of math results that LMs have recently been proving tend to have worked in that area for a long time. 3/

Now, I think that students who pursued math tended to get better at generalizing within math, but my sense is that is primarily by *studying* it and seeing the repeated structures and strategies (indeed some classes explicitly teach that) 2/

I would agree that AIs performance drops OOD. But these arguments seem to often presuppose that human performance doesn't, and I think that's just clearly wrong. I used to TA calc (at a good school) and it was pretty clear that people strongly over it to the problems they saw in class / the book 1/

I agree that they are different learning processes and efficiencies. I am not sure why that should change questions of generalization, which we can already demonstrate at small scales. (And of course, it's worth noting that we train models to be competent across many more domains than any person.)

New blog post! I wrote about why I think it's useful to use metaphorical (and anthropomorphic) terms to describe AI. I start by describing the role of metaphor in language and cognition, and how metaphors can help us to efficiently understand relational structure and make predictions 1/2

We spend a long time educating scientists in how to reason about science, even though we think humans can generalize. Generalizing is not easy.

Well surely if you don't think these models can do any interesting generalization out of distribution, data doesn't matter? Since all the interesting tasks in science will by definition always be out of distribution :)

Are you: * A cognitive science (or adjacent) PhD (current or graduated, or equivalent research exp.)? * Interested in studying LMs abilities for conceptual reasoning about cog sci, and helping to improve them? If so, apply to our new conceptual reasoning fellows program! Link below: