Students reading code with an AI assistant carried more cognitive load and did not report it

Twenty students from two Java courses each read six Java snippets of about 30, 45 and 70 lines, half with a Gemini chat window open on a second monitor and half without, then answered four comprehension questions per snippet without the assistant. Throughout, a small motor on the shoulder or arm vibrated every three to five seconds and the participant pressed a button in response. Slower responses indicate that the main task is taking more attention. The method is an ISO-standardized detection-response task, and the authors say that to their knowledge it has not been applied in software engineering before.

With the assistant, reaction times were 11.2% higher overall (p < .001): 12.7% higher while reading, and still 6.9% higher during the quiz, when the assistant was no longer available. Self-reported workload (NASA-TLX) and task difficulty (SEQ) did not differ significantly between conditions. Neither did comprehension: mean quiz scores were 3.22 of 4 with the assistant and 3.23 without. Scores clustered near the maximum, and the authors read the result as "an absence of evidence for a benefit, not as evidence that AI confers none." Java experience did not change the effect. In open-text answers, roughly a quarter of participants described letting the assistant's explanation replace reading the code.

The sample is 20 students, the code is short, and the assistant sat in a separate browser window, not in an IDE. The authors also note that higher load is not necessarily harmful; it can reflect engagement.

Why it matters: If you judge an AI tool by asking developers how hard the work felt, this experiment is a caution: that question missed an increase the instrument measured. The authors recommend adding objective load and performance measures. The study covers students with a chat window, not professionals reviewing agent-written changes.