Extraordinary claims require extraordinary evidence, and I am not the one making the extraordinary claim, so it is not my job to specify what the evidence should be. >>

It's always struck me that models that represent related concepts near each other in latent space could help w/ derivations. RL needed for more reliable step-by-step, tho. And there'd still be a lot of incorrect derivations...(Do we have stats on performance without external tools/verifiers?)

*I should clarify that neural networks *were* a part of the proposal -- NNs were really interesting at the time! I was pointing out the insights that were lost from others, like feedback mechanisms, that only made it in much later.

😞

...every person invited to the founding workshop was a man. Women doing early work (eg Margaret Masterman) were left out of AI's origin from day one. For everyone in AI who's been involved in "us vs them", I do feel it's important to recognize (among other things) this history. /end

They hated Norbert Wiener (McCarthy especially, so I hear). This act of gatekeeping pushed out work on, among other things, neural networks. As well as the women in the space…3/

It was named “Artificial Intelligence” in 1955 to get funding for a 1956 workshop. The naming was an act of branding: Something that sounded exciting & positive, without needing to be clearly defined. It was also a “line in the sand” to exclude Norbert Wiener, of cybernetics fame, in part because…2/

Oh, it was 100% born toxic! The first AI workshop with McCarthy was named as such do distinguish it from cybernetics, because McCarthy hated Norbert Wiener. It was quite literally an act of “us vs them” in the automata studies of the time.

October Dahlias, arguably the Best Dahlias

In the Biden administration, l led the team that developed what was arguably the first White House “AI safety” framework, the AI Bill of Rights. David Robinson, author of this forthright and sobering essay about risk culture at OpenAI and in Silicon Valley was on that team.

Did you consider the title “attention is REALLY all you need”, hehe.

You could see it as: 😏,🐇,🥸,🪶,🥳 is just about as likely as 😜,🐇,🥸,🪶,😵‍💫 Even with identical-looking tokens at the start at the end, it’s still just a sequence of tokens (and perhaps to your paper’s point, even if they look identical at the surface level, they’re not, due in part to their positions)

Idk, I’ll have to read your paper, but it’s not like even the “same number” on input and output is the same. “101”,”+”,”9”,”=“,”110” being just about as likely as “100”,”+”,”9”,”=“,”109” doesn’t seem unintuitive to me at the form level. It just requires removing my own intuitions about math.😅

Oh, I see your paper is still asserting the importance of positional information. Yeah that makes sense to me!

I mean, *you* carry but that doesn’t mean *it* has to carry. Cool work, I’ll check it out!

Remember these are tokenized with extremely long context windows, with each token associated to a position. So numbers in different positions lead to different outputs. Like, for the number “123456789101”, it’s “1 given ‘1’ 11 positions prior and ‘2’ 10 positions prior and ‘3’ 9 prior” etc