A new benchmark for deep learning based affinity prediction: Solving the inter-protein scoring noise problem

Abstract

The incredible advances in deep learning architectures have not only led to successful protein structure prediction methods, but also to many interesting new methods related to protein-ligand docking and co-folding. The most recent biomolecular foundation model, Boltz-2, even claimed to approach the performance of free-energy perturbation (FEP) in estimating small molecule–protein binding affinity. However, there is still an ongoing debate about the capabilities of such models regarding generalisation or memorisation. A comprehensive assessment remains difficult due to the lack of demanding and relevant benchmark data sets. Here, we present a new approach to benchmark affinity prediction based on the interprotein scoring noise problem of classical scoring functions. This well-known phenomenon describes the problem that classical scoring functions are capable of enriching active molecules for a protein target but are unable to identify the correct protein target for a given active molecule. This is due to scoring variation between different binding pockets, which limits the ranking capabilities for target identification. In contrast, a successful and reliable binding affinity prediction method should be able to identify the correct protein target and differentiate between binders and non-binders. We therefore developed a new benchmark set for target identification based on the LIT-PCBA and tested whether Boltz-2 can correctly identify the target of active molecules. Our study highlights that, while the affinity module of Boltz-2 seems promising, it is still not able to identify the correct protein target by correctly predicting a higher binding affinity in comparison to the decoy targets. This indicates that a generalisation of protein-ligand interactions is still not achieved, which is in line with other benchmarks that indicate a memorizing effect. Such a benchmark can be seen as the next major hurdle to successful deep-learning-based affinity prediction using protein-ligand complexes. Any model or method truly capable of accurate binding affinity prediction should perform well on target-prediction benchmark tasks