In a study of Claude Code migrating ML projects from GPUs to TPUs, 49% of failed execution attempts raised no error

The authors built a dataset of 148 PyTorch research projects and had Claude Code, running Claude Opus 5.5, migrate ten of them to Google Cloud TPUs. They then checked each migration themselves instead of relying on the agent's report. Eight of the ten preserved the project's goal, with 2.4 human interventions per migration on average. Of 70 execution attempts, 53 failed, and 26 of those 53 failures raised no error. The most common cause was lazy-tensor execution, which can recompile repeatedly or force host synchronization while the program keeps running. Only seven of the 148 projects had author-written tests.

The sharpest case: on a four-chip run, the agent reported success while accuracy fell from 86.53% to 84.89%. The authors trace it to a pinned training library that clipped gradients before averaging them across chips. The agent's own diagnosis blamed something else. Seven of the nine completed TPU runs used under 25% of the accelerator, and agent token costs were US$2 to US$23 per project.

These are the authors' measurements on one agent, one single-chip TPU target (the four-chip run is a case study), PyTorch only, ten projects that fit on one chip, and a single seed per run. They say differences under 1% cannot be told apart from run-to-run variation. We read the paper's text through the threats-to-validity section and did not open the replication package.

Why it matters: The agent's final claims did not exceed the authors' assessment on the nine completed projects, and still one case was a false success. If your agent migrates code or hardware targets, the check that failed here is the one worth building: a reference run on the original setup, repeated under the configuration you intend to ship.