Imagine handing a program to an AI and asking it to make the code run faster. The change looks like a win: the computer finishes sooner, and the code appears more efficient. But speed is only one part of the bill. When researchers measured both the energy used to produce AI-refined code and the energy used to run it, a seemingly obvious bargain often fell apart: faster-looking code was usually no greener than a strong human-written solution.
The team tested 11 open-source language models on 1,000 coding problems from EffiBench, a collection of LeetCode tasks paired with highly rated human solutions. Each model generated code, then tried to improve it through five rounds of self-refinement. After each round, the system received feedback about execution time and memory use. The researchers then measured energy on the computing hardware, rather than assuming that shorter runtime meant lower energy use.
That distinction matters because the optimization process has two separate costs. First, the model uses energy to generate and revise code. Second, the resulting program uses energy every time it runs. A revision pays off environmentally only if its repeated runtime savings eventually outweigh the energy invested in creating it. And, for a fair comparison, the researchers measured the AI’s code against the benchmark’s human-written reference—not just against the AI’s own earlier draft.
The headline result overturned the expectation. In 42 of 44 combinations of model and decoding settings, the refined code used more energy per problem than the human reference. The excess ranged from 4% to 886%. Only two combinations produced code that used less energy on average, and those savings were small. The study’s energy numbers cover particular models, settings, and benchmark problems; they are not a universal verdict on AI coding tools. But they show that “optimized” can mean faster without meaning more energy-efficient.
Why can those goals diverge? The refinement system was designed to improve execution time and memory use; it did not select code based on energy. The researchers found that more than half of the tested configurations actually regressed in execution energy over the refinement process, compared with their initial code. A program can finish sooner while drawing power differently or doing work in a way that consumes more energy. Runtime alone cannot tell you which outcome you got. The experiment measured that mismatch, though it did not establish one cause for every case.
The upfront cost made the rare savings look more distant still. Across five-round optimization runs covering all 1,000 problems, energy use ranged from 0.79 to 8.68 MJ across the tested model-and-setting pairs. For one of the two combinations that beat the human baseline on execution energy, the researchers estimated that it would take about 3.4 × 106 executions of a representative problem to recover the energy spent generating the optimized code. The other needed about 6.4 × 107. For the remaining 42 combinations, there was no break-even point against the human reference: their code consumed more energy each time it ran.
There was another surprise in the model comparison. A 35-billion-parameter Qwen model built with a mixture-of-experts architecture used 3.12 MJ in the default optimization setup, while a smaller, dense 27-billion-parameter Qwen model used 8.58 MJ. The larger model activates only a subset of its parameters for each token. Its correctness score was close to the dense model’s, too: 78.6% compared with 81.0%. In this experiment, how a model was built mattered more for energy use than the simple count of its total parameters.
One tempting shortcut—limiting the model’s output length—also came with a tradeoff. Cutting the maximum output from 2,048 tokens to 1,024 reduced inference energy by up to 69% in the tested configurations. But the shorter budget sharply reduced correctness for several models. Saving energy while producing code that fails its tests is not a useful optimization. The result underlines that energy, speed, and correctness need to be judged together, not treated as interchangeable signs of progress.
The boundaries of the experiment are important. These were Python solutions to short, algorithmically demanding tasks, each completing in under a second. The human reference solutions were top-rated competitive-programming submissions, a tougher comparison than ordinary workplace code might present. The study used open-source models running on instrumented local hardware; it did not measure commercial systems served through APIs. Longer-running software, other programming languages, and less optimized human code could change the energy balance.
Still, the central lesson reaches beyond a leaderboard. When AI is asked to improve software, its success cannot be judged only by whether the new version runs faster or uses less memory. The energy spent asking for the improvement belongs in the accounting, too. If code runs millions of times, even a small per-run saving might matter—but this study found that such savings were rare against its strong human baseline. Sometimes the efficient move is not to ask the model for another revision, but to measure what the last one actually cost.
