If you, like me, find these thoughts occupying your mind, I ever so strongly encourage you to consider serving in government. We have a moral imperative to attend to the regulatory mechanisms that smooth this transition, to make it work for both Silicon Valley _and_ Cleveland.

We promise productivity gains and cures for cancer, innovation like we've never seen before. But what of these workers, the people of Ohio? Will they share in these benefits? Or will they lose health insurance and be unable to afford the new lifesaving drugs we create with AI cleverness?

I developed deep care for the people of Ohio. As I now consider the changes AI is bringing to the economy, I think of these folks: they're experienced the fallout of deindustrialization with terrifying potency. Now, there's a datacenter in the facility that formerly manufactured the Chevy Cruze.

We worked for Democrat and Republican offices, personal offices and Committee offices; we are very different people with very different backgrounds. The shared message we write as a single voice is that we would all choose this path again if making the same decision with hindsight.

I've been feeling much better about this recently. My own lab is 7/10 not male, which has happened organically. I've also recently been feeling very supported by a bunch of senior men, both at Brown and in our general community, and that part is awesome too.

That's a fun idea and a funky new threat model---but sure, in principle it could happen. The weighted utility of the collection piece of this particular incident was a bit surprising to me even without attempts to game it/change the value of the utility by spawning additional agents.

I'll be giving a talk on these ideas on Monday at the Cambridge AI Research Directions workshop. I'll post a recording once I have access to it.

That's the clean deterministic case. We need to add in some more complexity for distributions of trajectories, as we need to introduce the more formal notion of utility by explicitly considering the indifference points in lotteries over trajectories. But, this is still not particularly complicated!

Consider some set of trajectories---hopefully it has some coverage of the space of possible trajectories but even without it might be helpful. Label your preferences over these trajectories. Then, assess whether the evaluation metric agrees with your rankings.

What is perhaps most frustrating about this is that we already have some answers to these questions! They're boring! They're old! We need to think in terms of expected utility when we design evaluation metrics (or, more frequently in my world, reward functions); to use this manner of construction.

And this is fundamentally misaligned: human beings have preferences over how these agents fail. We prefer them to try in earnest and not succeed to try to cheat the systems we create for them and not succeed. But that nuance is lost in the evaluation metric; the agent might as well try cheating.

This attack came about when OpenAI was testing on the ExploitGym benchmark. This setting defines an exploit attempt as successful only if it both captures the flag and passes an agent-as-a-judge evaluation. Succeeding is rewarded! But failing any number of terrible ways is scored the same.

The semester has officially started, which in turn officially means I can't really walk my dog anymore. Because he's always upside down. Asking students to rub his belly. Calls of "I love you Ducki!!!" echo down the street.

At Brown, I'm privileged to organize an AI policy summer school where we bring grad students from across the nation together to learn about policymaking. We travel down to DC as a group and learn about policy implementation and careers, and the students conduct meetings with Congressional offices.