A protein begins as a string of amino acids. Fold that string into a three-dimensional shape, and the result may be a working molecule—or a biological puzzle. AI models such as Boltz-1 can predict protein structures, but their internal calculations are harder to read than the structures they produce. Researchers Piotr Jedryszek and Oliver M. Crook set out to find out whether those hidden calculations contain recognizable biological clues—and how to tell a real clue from a convincing-sounding guess.
Their work brings two kinds of investigation together. In protein biology, researchers look for patterns such as a residue’s position in a structure or its role at a functional site. In AI interpretability, researchers try to break a network’s tangled internal activity into simpler features. The team examined features from Boltz-1’s Pairformer trunk, a part of the model that processes information used in predicting biomolecular structures. Their question was not just whether an AI could describe a feature, but whether its description survived a biological check.
To make the tangle more manageable, the researchers used sparse autoencoders, tools that separate a model’s activations into simpler patterns, called latents. A latent might activate around a particular amino acid or in a recurring structural context. But seeing a pattern in one run is not enough: the pattern could be a quirk of how that particular dictionary was trained. The researchers’ first test borrowed an idea from reproducibility. They trained dictionaries independently, matched their features, and asked which patterns reappeared across runs. Features that matched more closely were deemed more stable.
That produced a three-part investigation. First, rank features by cross-run stability, before spending time on expensive evaluations. Second, test whether a feature’s examples actually hang together: show a judge four protein-sequence windows where the feature activates and one where it does not, then ask which window is the odd one out. Third, ask for a biological description separately, translate that description into a prediction, and check it against curated protein annotations. The description does not get to coach the odd-one-out judge. That separation helps distinguish “these examples share a pattern” from “this label sounds plausible.”
Even the test of the test mattered. In early trials, a restrictive response limit made the judge perform almost at chance on known amino-acid controls. The team traced the problem to the evaluation setup: the judge had too little room to reason through the examples. Raising the output allowance from 16 to 512 tokens lifted accuracy on those controls from 0.17 to 0.97. Prompt wording and metadata made much smaller differences. It was a reminder that an apparent failure to recognize biology might instead be a failure of the measuring instrument.
With the evaluation calibrated, the researchers screened 150 low-frequency features; 141 passed a preliminary density check. Seventeen met the study’s stability cutoff, and nine of those passed the interpretability test. In this sample, the stability filter found interpretable features using about 4.4 times fewer evaluations per finding than checking the whole panel. Measured cost per finding fell by 5.2 times. In a separate analysis using annotated concepts, the same stability cutoff nearly doubled the rate of interpretable features. These are results from a focused study of one model’s internal features, not a general guarantee that stable features are always meaningful.
The biological cross-check added another safeguard. A description such as “fires on zinc-coordination residues” implies a testable prediction: the feature should be enriched at annotated zinc-binding sites, even when the comparison is restricted to the relevant amino acids. All 17 interpretable features’ residue descriptions passed the researchers’ annotation check; 12 of the 16 testable claims about context did too. One particularly striking feature marked the histidine in a kinase’s His-Arg-Asp, or HRD, motif. It matched nine of 23 annotated sites and was 114 times enriched for that motif among histidines. That is evidence that the feature’s activity tracks a real biological pattern—not proof that Boltz-1 uses it as a causal step in its computation.
Here the story takes an unexpected turn. The filter’s most repeatable clues were often structural: for example, a particular amino acid near a helix cap or linker. Some of the functional clues the team found—features associated with catalytic or metal-binding sites—were less stable across training runs. Of the features they examined, the stability filter kept eight of nine that detected a residue at a structural position, but lost four of five that named a functional site and all three that named a compositional region. In other words, a method that saves time can quietly favor the kinds of biology it already sees most consistently.
Why did those functional features appear at all? The paper cannot settle that question. They might be signals the model genuinely uses but that the stability screen overlooks. They might instead be patterns present in the data that the autoencoder can detect, without being part of Boltz-1’s working computational basis. Or they could be real features that do not fit the simple, repeatable directions the researchers measured. The sample of functional motifs was small, and the authors call for larger tests and causal interventions to tell these possibilities apart.
That uncertainty shapes the practical lesson. The researchers recommend using stability to rank features for attention, not to discard every unstable one. A biological label should also be treated as a hypothesis, not a discovery, until it makes a prediction that can be checked against independent evidence. For already annotated protein patterns, that check can happen in data; truly novel claims may need laboratory experiments. The bridge between AI and biology, then, is not a machine that explains itself. It is a disciplined chain of questions: Does the pattern recur? Can it be recognized without being named? Does its proposed meaning match the biology? And what important clue might the filter have left behind?
