Possibly! But there's a big hurdle to be solved before we can do that, which is that the method so far has only been applied within constrained input domains. The smaller model works as well as the larger one in these constrained domains, but we don't yet have methods that distill more broadly

My sister and I saw the Lion King in mid-September, and I just realized that this was perfect timing: the pride goes before the fall

A nor'easter is a storm with winds from the northeast So you might think a sou'wester would be a storm with winds from the southwest. But it's not!! It's a type of hat!! Language strikes again

Had a great time speaking at NYU today - there's an incredible collection of people there working on computational CogSci & AI!

You're welcome, and thanks for the question - it's an important one!

But that's not the case here: Once we fix a structural hypothesis ("role scheme"), our TPR approximation model is highly constrained, as it's bilinear. There's no guarantee that it can approximate anything, so when it *does* succeed, it's strong evidence that the hypothesized TPR structure is there

One more point: Suppose our approximation model were a universal function approximator, like a standard neural net. Then, our approximation wouldn't mean much, since we could approximate anything, even something that didn't use the structure we hypothesized

Second, once we have a TPR approximation to an LLM, we can take the LLM's representations & edit them as if they are a symbolic structure, and the edits change the LLM's behavior as intended. This editability gives more fine-grained evidence that the symbolic structure is there in the LLM

Why do I feel comfortable concluding Point 3? I.e., how can we conclude "equivalence" from data that merely show "approximation"? First, the way we evaluate an approximation is by seeing if it yields the same behavior as the original. This establishes a type of equivalence (behavioral equivalence)

Yeah, good q! The argument is: 1. A TPR is a symbolic structure 2. Thus, if a TPR is equivalent to the LLM rep, then the LLM rep has symbolic structure 3. Our results show that the TPR is equivalent to the LLM rep Point 3 is certainly debatable, but I do feel comfortable concluding it from our data

That said, it might well be the case that it's easier for the more recent architectures to discover TPR structure than GPT-2. It could be interesting to look at the training trajectory to see if TPR structure arises faster in some models than in others!

Great question! GPT-2-XL approximated about as well as the architectures in that figure. So, my intuition is that the TPR structure arises more out of top-down pressures (i.e., what the representations are used for) rather than bottom-up pressures (i.e., architectural structure).

Thank you for the kind words!

But I do think there could be something there - will keep thinking about it!

E.g., the embedding dimensionality is a free parameter, and your choice for that dimensionality might really influence whether you get linear independence. So the linear independence route might be too dependent on properties of the analysis, rather than the model being analyzed.

That said, that intuition might be hard to actually implement, because the representations of roles that we extract in our approach depend somewhat on hyperparameters.

Ooh I love this idea! The TPR formalism definitely makes predictions in this direction: TPRs guarantee perfect decoding only if the embeddings of roles are linearly independent, but if they're not linearly independent, you'll get intrusions.