Mechanistic Interpretability of Fine-Tuned Protein Language Models for Nanobody Thermostability Prediction

Abstract

Abstract Motivation While Protein Language Models (PLMs) fine-tuned on biophysical data achieve high predictive accuracy, the physical principles underlying their predictions remain obscure. Deciphering these representations offers a unique opportunity to not only interpret model decisions but also to discover novel biophysical insights governing protein properties. Here, we present a framework using Sparse Autoencoders (SAEs) to extract mechanistic knowledge from PLMs fine-tuned for nanobody thermo-stability. Results We fine-tuned the ESM-2 model on the nanobody thermostability dataset, achieving superior performance compared to significantly larger state-of-the-art models. SAE analysis successfully decomposed the model’s dense embeddings into sparse, interpretable features without loss of predictive accuracy. We characterized these features through both global and local analyses. Global analysis revealed that stabilizing features predominantly activate on the molecular surface, optimizing charge distributions, whereas destabilizing features map to the hydrophobic core, reflecting packing defects. Local analysis identified specific interactions, including known factors like the VHH-tetrad and critical disulfide bonds, as well as novel stabilizing candidates. Free Energy Perturbation calculations confirmed that these novel candidates exhibit structural compatibility, avoiding the severe destabilization seen in neighboring residues. By revealing these multi-scale biophysical rules, our approach demonstrates that interpreting fine-tuned PLMs provides a physics-grounded guide for rational protein engineering. Availability The data and source code of the proposed method are available at GitHub ( https://github.com/matsunagalab/paper_nanobody-thermostability-sae ) and Zenodo (DOI: 10.5281/zenodo.18012027). Contact Yasuhiro Matsunaga ( ymatsunaga@riken.jp ) Supplementary information Supplementary data are available at Bioinformatics online.