Dan Roberts

I am a Research Scientist at OpenAI, working on [redacted], and also a Visiting Scientist at the Center for Theoretical Physics at MIT. Previously, my research focused on the ways in which tools and perspectives from theoretical physics can be applied to AI.

I am a co-author of The Principles of Deep Learning Theory, co-authored with Sho Yaida, and based on research in collaboration with Boris Hanin. You can buy a copy in print from Amazon, directly from Cambridge University Press, or download a free draft from the arXiv!

If that sounds interesting, but perhaps also a bit much, you could instead check out my essays, Black Holes and the Intelligence Explosion and Why is AI hard and Physics simple?, or listen to to my podcast interview, What Physics Can Teach Us About AI.

I’m also a member of the advisory committee for the AIMO Prize, and I’m very excited about the development of AI models that can reason, mathematically or otherwise.

More broadly, I am interested in the interplay between physics and computation. My work in theoretical physics has focused on the relationship between black holes, quantum chaos, computational complexity, randomness, and how the laws of physics are related to fundamental limits of computation.

Rounding out my CV in natural language, most recently I was an AI Fellow at Sequoia Capital. While I was at Sequoia I tried and failed to found a new AI company, which would have been a sequel to Diffeo, where I was Co-Founder and Chief Technology Officer. Before that, I was a Principal Researcher at Salesforce, having arrived via the Diffeo acquisition. Even before that, I was a Research Scientist at Facebook AI Research in NYC. Yet further into the past, I was a postdoc in the School of Natural Sciences at the Institute for Advanced Study in Princeton, NJ. I completed my Ph.D. at the Center for Theoretical Physics at MIT, funded by a Hertz Foundation Fellowship and the NDSEG. Prior to that, I was a Marshall Scholar in the UK. While there, I read for Part III of the Mathematical Tripos at Cambridge and then studied quantum information at Oxford. In a previous life (undergrad), I worked on invisibility cloaks (metamaterials and transformation optics) with David R. Smith.

My name is very common, so I try my best to publish under more parts of my full name Daniel A. Roberts—which I never go by and which (unfortunately) also happens to be very common.

Artificial Intelligence

As an AI researcher at FAIR, Diffeo, Salesforce, Sequoia, and MIT, I focused on applying tools from theoretical physics to gain insight into machine learning and artificial intelligence. Recently, I worked on understanding neural scaling laws, pruning LLMs, robust learning, stochastic gradient-based optimization, causality, and the large-width expansion for deep learning.

In 2012, I co-founded Diffeo, a startup focused on collaborative machine intelligence. As part of Diffeo Labs, I co-organized a track for the Text Retrieval Conference (TREC) called Knowledge Base Acceleration. In 2019, Diffeo was acquired by Salesforce, which explains why a bunch of these hyperlinks no longer make any sense.

I once tried to apply machine learning to particle physics.

Theoretical Physics

I’m interested in black holes. I’m also interested in quantum information theory. Luckily, via the gauge/gravity duality or holography, these two subjects are intricately tied together.

Some of my work focuses on what happens when something falls into a black hole (in anti-de Sitter space). The black hole will very quickly scramble (but not destroy) the information. Black holes are thermal systems, and this is actually a manifestation of the well-known butterfly effect. We can try to think about this process in terms of its computational complexity, or we can study it as a distinguishing feature of quantum chaos.

Publications

Talks on YouTube

A portrayal of what it might look like to watch my talk on YouTube.

IAS Physics Talk – The Principles of Deep Learning Theory
Defense – CHAOS, Holography, And Other Science
Strings 2015 – The butterfly effect in spin chains and 2d CFT

Podcast Interviews

Sequoia’s Training Data – What Physics Can Teach Us About AI

“Press”

Advancing AI theory with a first-principles understanding of deep neural networks – Blog post by Sho Yaida announcing PDLT.
How Hertz Helped an AI Company Take Shape – Writeup of the Diffeo story, from founding to acquisition by Salesforce.
Black Holes Produce Complexity Fastest – Viewpoint in the APS journal Physics (which is very nicely written) on Complexity Equals Action.
Complexity growth – Research highlight in Nature Physics (which is two paragraphs—one and a half of which are behind a paywall—and is unfortunately incomprehensible) on the same work.

Other Writings

Thesis – stapled papers + jokes = PhD