We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle" · Andrej Karpathy · X, August 2, 2026, 3:00 AM UTC. The post states no affiliation and this publication records none for him.
Karpathy runs an agent for two hours, then names what it could not do
Machine-readable
Download Markdown
Karpathy gave Claude Opus 5 the first paragraph of The Lord of the Rings, a 1M token budget (about $10) and asked for a three.js render. In his words: "Opus went off for ~2 hours and wrote 5500 lines of code that (procedurally) rendered the story." He frames the economics as a phase change rather than a speedup: "no one in their right mind would ever spend the time to write something this custom but LLMs have all the stamina and patience in the world, so it's an example where we go from 'no one would ever do this' to 'sure, why not, it's ~free'." The last paragraph earns the lead. "The domain of worlds/games exposes a weakness in LLMs: they can't easily audit their work because they aren't able to efficiently and natively perceive videos or play games within them. Here, Opus 5 had to very slowly and painstakingly take screenshots at different points, and it messed up a few times and created a bunch of jank." Two caveats the surrounding coverage has not carried. The run was not unattended end to end: asked how the audio was made, Karpathy answered "Eleven Labs for the audio. LLMs can easily use the APIs (here I did that part manually because I felt picky about the voice)." And the output is inspectable, because a follow-up post published the source at karpathy.ai/lotr-movie, "forkable etc." Figures read at 20:15 EDT, about 17 hours after posting: 23,153 likes, 1,743 reposts, 1,197 replies, 2,783,444 views. The Hacker News thread was the board's top item at 407 points and 321 comments at 20:20 EDT, up from 319 and 260 at 18:10 EDT.
Why it matters: A lights-out factory needs the agent to close its own verification loop, and here the person running the demo reports that on this task it largely could not. Karpathy scopes the weakness to worlds, games and perceiving video, so take the scope as he states it. What carries past the scope is the mechanism: the agent's only channel for checking its own output was screenshots it took itself, and that channel was slow and wrong several times. Where your harness verifies through tests and a compiler, that channel is cheap and reliable. Where it verifies through something the agent has to look at, this is one dated account of the cost.