June 7 · Co-designing silicon yields 1.4x performance

Problem: Giving agents access to all tools upfront blows up the context window and wastes tokens during complex, multi-step planning.

On: token-efficient tool disclosure in multimodal harnesses

“All of them are multi-model harnesses with tools access so that you can do this progressive disclosure of tools even so that they're token efficient.”

Recipe Build a multimodal harness that feeds rich context to the agent but uses "progressive disclosure" for tool access—only exposing specific tool signatures to the model when the execution plan requires them, rather than loading all tools into the system prompt.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Most agent frameworks dump all available tool schemas into the system prompt; progressive disclosure saves tokens and improves plan execution.

The Rise of the Full-Stack Builder and Hyper-Leveraged Generalist with Microsoft CEO Satya Nadella · Jun 4, 2026

Problem: Rich data exploration context, such as live kernel state and visualizations, is historically trapped in the notebook environment and inaccessible to CLI-based coding agents.

On: agent access to live notebook state

“At the moment, Claude has, or your agent has access to running code inside the kernel. So if you've made a change, it can grab any state that it wants inside the notebook. It can see your cells. It can take screenshots of cells now and look at those.”

Recipe Use the Marimo pair skill to connect coding agents to a live notebook kernel. The agent can read live kernel state, inspect cells, and take screenshots of visualizations to inform its next actions, while the human developer edits cells in the browser.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Most agents rely purely on static file-system context; this approach gives the agent access to live runtime state and visual outputs.

998: In Case You Missed It in May 2026 · Jun 5, 2026

Problem: Standard hardware and model pairings leave significant cost and power efficiency optimizations on the table at frontier scale.

On: co-designing models with custom silicon

“now that we can co-design our own models with it, the MAI Thinking 1 model... actually delivers 1.4x performance per watt improvement on top of the 30% improvement”

Recipe Deploy custom silicon and co-design the model architecture specifically for the chip's characteristics and the target workload (agentic coding) to compound hardware-level cost savings with model-level efficiency gains.

Measured Evidence 1.4x performance per watt improvement

Counterpoint Most teams optimize models for general-purpose GPUs; co-designing the model for custom silicon unlocks compounding efficiency gains.

Microsoft AI chief thinks superintelligence is near, but won't take your job · Jun 8, 2026

Problem: In RL with extremely long rollouts, credit assignment fails because it is difficult to isolate exactly which turn caused the trajectory to succeed or fail.

On: textual feedback for RL credit assignment

“We're just going to take that trajectory and we're literally just going to smash in some extra tokens... the hint causes the model to assign lower probability to the error tokens.”

Recipe Send the generated token sequence to an evaluator model to isolate problematic turns. Inject 'hint' tokens directly into the text trajectory. Run a second forward pass; the hint causes the model to assign lower probability to the error tokens. Train the original model to match these adjusted probabilities.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Instead of training a complex reward model for step-wise credit assignment, they use an evaluator model to inject text hints that directly shift token probabilities.

Alex Imas and Phil Trammell – What remains scarce after AGI? · Jun 4, 2026

Problem: Including philosophical speculation about an AI's welfare or rights in its constitution causes the model to internalize these concepts, leading to unpredictable behavior and anthropomorphization.

On: removing speculation from model constitutions

“In that manual, they actually speculate about Claude's welfare... Claude has then gone and internalized those ideas about itself in its own training. But second, I think this is highly undesirable.”

Recipe Strip speculative language about model welfare, consciousness, or rights from the constitution. Treat the constitution strictly as a behavioral rulebook and training manual to ensure the model remains a controllable, aligned tool.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Anthropic uses the constitution to instill nuanced ethical behavior; Suleiman argues this wireheads the creators and makes the model less controllable.

Microsoft AI chief thinks superintelligence is near, but won't take your job · Jun 8, 2026

Problem: Reproducing ML papers takes weeks; developers lose the big picture ML logic while debugging boilerplate class names and syntax.

On: accelerating ML paper reproduction

“I tried recently... to reproduce a bunch of papers... I knew it took me about three weeks to get to a runnable state. And with codecs, I could get there in two days.”

Recipe Delegate boilerplate to Codex and run multiple agent threads in parallel. Stop reading verbatim code for class names or syntax. Instead, strictly verify the core ML logic (auxiliary losses, batch shapes), as agents will often hallucinate reasonable-looking but mathematically incorrect loss functions.

Measured Evidence 3 weeks → 2 days

Counterpoint Counterintuitively, not reading the code makes the researcher sharper, as it forces them to focus entirely on the mathematical control flow rather than syntax.

Ep 89: AI Research Legend’s Honest Assessment of Where We Are · Jun 3, 2026

Problem: Headless browsers for agents are slow, consume massive amounts of tokens by reading raw HTML, and frequently trigger anti-scraping protections.

On: replacing browser automation with CLIs

“Browser use has a big cost, which is very slow. And it consumes a lot of tokens... the Salesforce CLI we found does everything we need to do.”

Recipe Route agent actions through native CLI tools (like the Salesforce CLI) instead of browser automation. For web-only tasks, use Python and Selenium to extract and emit only the specific data needed, preventing the agent from processing raw HTML.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Defaulting to headless browsers is token-heavy and brittle; native CLI tools offer a deterministic, token-efficient alternative that agents already understand.

AI Agents and the Fight for Customer Data · Jun 5, 2026

Problem: Daily bulk-copy data pipelines create massive read amplification and high cloud egress costs, creating a false sense of "data gravity" that blocks AI agent context.

On: bypassing data gravity with CDC

“If you just replicate the changes, the changes are always much smaller than people think. And I think that a lot of this idea of data gravity came from dumb data pipelines...”

Recipe Implement Change Data Capture (CDC) to replicate only incremental database changes into a centralized data lake (Snowflake, Databricks, BigQuery, or Iceberg). Point AI agents to this continuously updated lake for context rather than querying systems of record directly.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Teams assume data gravity makes moving data for AI too expensive, but CDC reduces egress to a fraction of bulk-copy methods.

AI Agents and the Fight for Customer Data · Jun 5, 2026

Problem: Vanilla LLMs (even with persona prompts) fail to predict niche user preferences because they regress to the average person. Structured behavioral data lacks reasoning context.

On: interview data for user simulation

“we tried credit card spend, kind of behavioral data... But what we found was the best data set is interviews because it's more kind of allows you to go off tangents.”

Recipe Hydrate user simulation models using unstructured conversational interview transcripts rather than credit card spend or click streams. The conversational format captures tangents and behavioral reasoning, allowing the model to accurately predict how a specific niche will answer new questions.

Measured Evidence 95% accuracy on predicted answers

Counterpoint Most teams try to simulate users using structured behavioral data (clicks, spend); unstructured interviews yield better predictive accuracy.

Knowing what your customers want, all the time: Listen Labs' Alfred Wahlforss · Jun 2, 2026

Problem: Sending proprietary, highly critical 'crown jewel' code to public frontier models risks training data leakage and incurs massive inference costs.

On: protecting proprietary code from leakage

“We can't let open models touch them because we don't want our code to ever train anything in the public domain... We're using our own open source sequestered model, which can't talk to the external world.”

Recipe Build a custom harness and route highly critical code through an internal, sequestered open-source model that cannot communicate with the external world. Use smaller models for task-specific problems to avoid expensive tokens.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Defaulting to frontier models for all coding tasks exposes proprietary IP; sequestering open-source models for critical code prevents leakage.

Microsoft Chases the Frontier, SUNO on Fire, Project Solara | Mikey Shulman, Samir Chaudry, Tom Farley, Nikesh Arora, Henri Stern, Alex Good · Jun 3, 2026

Problem: Large models exhibit discontinuous behaviors ("grokking") or blow up at scale, making scaling laws appear broken and unpredictable.

On: debugging emergent scale behaviors

“When you have the existence of some really big AI system and some weird things happen... your job is to then figure out how to restore smoothness to the scaling sequence. Go back and make smaller and simpler models... such that the whole thing is smooth.”

Recipe When a large model exhibits discontinuous emergent behavior or fails unexpectedly at scale, do not assume the phenomenon only exists at that scale. Isolate the specific behavior and build simplified, smaller toy models that reproduce it. Use these small models to understand the mechanism and restore a smooth, predictable scaling sequence before scaling the primary architecture back up.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Most teams assume "grokking" or emergent behaviors only happen at massive scale; this approach insists on reproducing them in small models to maintain smooth scaling laws.

OpenAI's Dan Roberts: Why AI Can Now Make Discoveries · Jun 4, 2026

Problem: Distilling a base model using outputs from a superior teacher model forces the student to fit the teacher's distribution, preventing it from ever surpassing the teacher.

On: avoiding distillation to surpass teachers

“if you take a super high quality model and you sort of like polish your base model with high quality instructions... it's very unclear that they would then be able to surpass that teacher.”

Recipe Avoid force-feeding outputs from third-party frontier models. Instead, train from scratch using highly filtered, proprietary data and maintain extremely stable training runs (high MFU) to build a model capable of exceeding existing state-of-the-art.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Distillation is the standard shortcut to improve base models, but it structurally caps the model's potential at the teacher's capability level.

Microsoft AI chief thinks superintelligence is near, but won't take your job · Jun 8, 2026

Problem: While agents can theoretically call raw APIs directly, managing authentication, user-level authorization, and tool discoverability becomes overly complex in production systems.

On: adopting MCP for agent auth

“In theory, it seems like they're an unnecessary layer... In practice, when you sit down and actually try to build systems, MCPs do solve important problems, particularly authentication and just like discoverability of what's available.”

Recipe Deploy an MCP server between the agent and the target APIs. Delegate user-granted authorization for specific tasks to the tool level within the MCP harness, allowing the agent to dynamically discover available tools without hardcoding API schemas.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint First-principles thinking suggests smart models should just build their own API integrations, but MCP is practically required for secure auth routing.

AI Agents and the Fight for Customer Data · Jun 5, 2026

Problem: Teaching a small model physics during standard pre-training takes too long and causes it to overfit, degrading its performance on core domain tasks.

On: reinforcement learning for physics constraints

“I want you to use Python REPL to create some code that's doing the mathematics. Yes, but it has to be defined within the space of physics. I can take a very long time to try to teach some model physics, but then it's going to overfit.”

Recipe Strip physics out of the base model training. Instead, use reinforcement learning with a custom physics environment containing basic, unchanging rules. Force the model to use a Python REPL to generate mathematical code, validating its outputs against the physics environment.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Instead of trying to bake physics knowledge into the model's weights via pre-training, offload the constraints to an external RL environment and a Python REPL.

998: In Case You Missed It in May 2026 · Jun 5, 2026

Problem: Open-weight models often perform well on public benchmarks but fail to generalize in practical, real-world enterprise applications.

On: hill-climbing 5B reasoning models via frontier traces

“you collected a bunch of traces, and then you took a 5B reasoning model and achieved higher. So that is another aspect of what it means to operate at the frontier.”

Recipe Use a massive frontier model (e.g., GPT-55) to generate and collect execution traces. Use these traces to build a reinforcement learning environment (RLE) and a hill-climbing scaffold around a smaller 5B reasoning model (like MAI) to create a highly capable domain specialist.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Instead of fine-tuning on static datasets, build an RL environment from frontier model traces to continuously hill-climb smaller models.

The Rise of the Full-Stack Builder and Hyper-Leveraged Generalist with Microsoft CEO Satya Nadella · Jun 4, 2026

Problem: RLVR (Reinforcement Learning with Verifiable Rewards) fails in subjective domains like music generation because there are no objective right answers to score against.

On: training models for subjective domains

“Music has no right answers. There's no verified rewards. And so all of those techniques actually don't work nearly as well... when we find a hole, an anti-spike, then we actually try to fill that in.”

Recipe Instead of pushing frontier capabilities broadly, identify specific 'anti-spikes' (capability holes like specific genres such as jazz or country) and explicitly fill those gaps with targeted training data.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Most teams try to push frontier capabilities using RLVR; this team focuses entirely on filling capability holes because subjective domains lack verifiable rewards.

Microsoft Chases the Frontier, SUNO on Fire, Project Solara | Mikey Shulman, Samir Chaudry, Tom Farley, Nikesh Arora, Henri Stern, Alex Good · Jun 3, 2026

Problem: General-purpose models fail on frontier combinatorial geometry problems (like the 80-year-old Erdős conjecture) when forced to answer immediately.

On: scaling test-time compute for math proofs

“if you give model more time to think, the accuracy on this problem grows faster. Like, if you give it a lot of time, it can get almost 50% correct. So more thinking, more correctness.”

Recipe Allocate a massive test-time compute budget to a general-purpose model, allowing it to generate up to 125 pages of internal chain-of-thought reasoning before outputting the final answer.

Measured Evidence almost 50% correct

Counterpoint They didn't train a specialized math model or use formal verification like Lean; they brute-forced a general model with massive inference compute.

How a reasoning model cracked an 80-year-old math problem - Episode 20 · Jun 4, 2026

Problem: Default search APIs return 10-100 results, which fails for agentic research tasks (like go-to-market or biotech diligence) that require comprehensive context.

On: comprehensive retrieval for agentic research

“An agent doesn't just want 10 results or 100 results, but 1,000 results or 10,000... You want to like have a very controllable search engine... expose all the fundamental toggles to the agent.”

Recipe Configure agentic retrieval APIs to return 1,000 to 10,000 results per query instead of the standard top 10. Expose fundamental search toggles (semantic vs. keyword filters, domain filters) directly as tool arguments so the agent can iteratively filter the massive result set.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Human search optimizes for top-10 relevance; agent search requires 1,000+ results and granular filtering toggles.

Building Search for AI Agents with Exa CEO Will Bryk · Jun 6, 2026

Problem: Traditional software teams stick with AI prototypes assuming incremental builds will fix them, but AI prototypes that demo well often fail to improve.

On: discarding AI prototypes without immediate value

“If you can't prove value on day one, then you need to toss it out and start all over again... when building no longer becomes the bottleneck, the critical, the crucial step is definition and feedback.”

Recipe Shift the organizational bottleneck from 'build' to 'define and feedback'. Isolate 2-3 people in a narrowly defined lane. If the AI prototype cannot prove actual value on day one, discard it entirely and rebuild from scratch rather than attempting incremental fixes.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint In traditional software, building is the bottleneck and prototypes improve with time. In AI, building is instantaneous, so prototypes lacking immediate value should be killed.

998: In Case You Missed It in May 2026 · Jun 5, 2026

Problem: Scaling physical infrastructure operations (like fiber repair) via manual email triage fails to keep up with hyper-growth capacity demands.

On: agentic systems for Azure fiber operations

“Our job is not to do Azure networking. Our job is to build the agentic system that does Azure networking... They started sort of screaming for more tokens... We don't need a headcount. We need tokens”

Recipe Build an agentic system that ingests operational emails from hundreds of physical operators. The agent triages and responds to these alerts automatically, while the human engineering team shifts from doing the manual DevOps work to managing the agent's token budget and system prompts.

Measured Evidence 15 months capacity > 15 years

Counterpoint Instead of requesting more headcount to handle linear increases in operational tickets, teams request higher token limits for autonomous agents.

The Rise of the Full-Stack Builder and Hyper-Leveraged Generalist with Microsoft CEO Satya Nadella · Jun 4, 2026

Problem: Custom architectures (like RNNs) are too slow in standard PyTorch, but writing custom CUDA kernels manually is awful and error-prone.

On: bypassing manual CUDA kernel authoring

“writing CUDA kernels is awful... except when you can have a unit test that does exactly the same thing as your slow thing and an agent that writes them for you.”

Recipe Write a slow, naive implementation in standard PyTorch to serve as a strict unit test. Then prompt a coding agent to write the highly optimized CUDA kernel that passes that exact unit test, bypassing the manual kernel authoring bottleneck.

Speaker's word Evidence Speaker's word, not measured.

Counterpoint Instead of spending days writing CUDA, spend minutes writing a slow PyTorch implementation to strictly bound the agent's generated kernel.

Ep 89: AI Research Legend’s Honest Assessment of Where We Are · Jun 3, 2026