Karpathy: understanding of LLM capability is narrowing into a sharp funnel
Karpathy’s napkin math: about 6 billion people have barely touched LLMs; 1–2 billion use free chat tiers and treat them as search; roughly 20 million see agents finish projects that used to take weeks; and only around 5,000 people inside labs see frontier systems at full strength. He attributes the split to problem scale and time horizon, capability peaking in verifiable digital domains, and the gap between free, paid, and internal access. The numbers are estimates, and the last step here is my inference, not his: most of the people shaping AI policy likely live in the funnel’s two widest tiers.
Thinking Machines: task expertise in RL beats the human bar on text-to-SQL
Thinking Machines, working with UIUC researchers, released ReViSQL-K2.6, and the method is mostly repair work: an audit found errors in 61.1% of BIRD Train instances (yielding a cleaned BIRD-Platinum), and semantic-equivalence checking with VeriEQL showed 32.8% of execution-match positive rewards were actually wrong. After fixing data and rewards, the model hit 92.97% accuracy with 16-sample self-consistency at $0.56 per task, which the team calls, to their knowledge, the first result past the 92.96% human benchmark. For anyone doing RL post-training, the point is that the gains came from cleaning data and verifying rewards; the RL objective itself (CISPO) already existed.
An agent fleet decompiled a full FPS game, reconstructing 99% of functions
Developer momo5502 ran over a dozen agents (a mix of Claude and Codex models) to decompile a first-person shooter back into readable code: an estimated 600–700 billion tokens total, 99% of functions reconstructed, 83% byte-exact against the original binary, and the game runs. The load-bearing piece wasn’t the token count but a byte-matching script that gave each function an objective PASS/FAIL signal. My read is that this is the same reason RLVR works: with verifiable ground truth, scale pays off. One more detail: two earlier posts about the project were taken down, and the author won’t even name the game, so outside pressure set the limits here before the technology did.
Hack The Box launches AI Range Enterprise Edition to test security agents against real jobs
Hack The Box opened AI Range to enterprise security teams on October 6, letting them test their own AI agents against specific roles, penetration tester and SOC analyst at launch, with per-role scores, pass/fail per environment, and performance over time. The gap it targets is concrete: agents go into production on a benchmark score or a vendor demo without ever being checked against the daily work. It also scores the humans, measuring whether security engineers can question an agent’s output and step in when needed.
Amazon and Microsoft drop NDAs in data center negotiations
TechCrunch’s Equity podcast picked up a shift: Amazon and Microsoft have stopped using NDAs when negotiating data center projects with local governments, against a backdrop of recent moratoriums, including a city-level one in San Francisco and a statewide pause in New York. The hosts’ read is worth hearing: the moratoriums are framed as “we need time to study this” rather than outright rejection, which leaves room to approve later; but dropping NDAs alone won’t convert opponents. Transparency is necessary, not sufficient.
Why voice AI hasn’t had its ChatGPT moment
PolyAI CTO Shawn Wen and Otter CMO Alex Gay point at the same bottleneck: speech recognition at the very front of the chain keeps missing key words, and as Gay puts it, “if your original transcription didn’t have the accuracy that you needed, all the follow-up actions that you have become flawed.” Wen adds that the next challenge is making reasoning fast enough for the conversation to feel natural. That moves the diagnosis from “the model isn’t smart enough” to “the input breaks at step one”: a bad transcript compounds through every step that follows.
Research radar
Agreement without alignment: LLM judges agree more, without getting closer to humans
The paper proposes a label-free procedure: find the rubric sub-criteria where LLM judges disagree most, and have a model rewrite only those clauses. On SummEval, inter-judge agreement rose by 0.069, but agreement with human raters showed no detectable change (Kendall τ difference of -0.007), and the other two datasets missed the preregistered thresholds entirely. Anyone running LLM-judge panels should read this: sharpening the rubric makes judges converge on each other, and convergence is not correctness.
One line for today: judges converging on each other is not the same as judging correctly, and I suspect the same holds for public consensus about AI formed on the funnel’s widest tiers; agreement reflects shared access, not shared understanding.