De novo molecular design with agentic AI: Integrating tree search and iterative refinement

Abstract

Agentic AI frameworks that couple language model generation with iterative tool feedback offer a promising paradigm for automating complex molecular design tasks, yet systematic comparisons of agentic design patterns, such as exploration-focused, refinement-focused, and hybrid, remain scarce. Here, we evaluate three such strategies applied to structure-based de novo molecular design. Language Agent Tree Search (LATS) frames generation as Monte Carlo Tree Search-guided exploration of chemical space. A Reflexion-inspired Self-Improvement (SI) loop critiques and revises candidates using cheminformatics filters and docking feedback. A Hybrid strategy uses LATS to generate diverse, high-quality seeds, which are then refined with SI. We benchmark these agentic design patterns across four protein targets (thrombin, transthyretin, HER2 kinase, and AKT1 kinase) using a unified AutoDock Vina protocol as a computational surrogate for binding affinity, complemented by analyses of drug-likeness, synthetic accessibility, novelty, and diversity. Across targets, the Hybrid strategy achieves the strongest mean docking performance (−12.17 kcal/mol on HER2) and remains competitive in best-case outcomes, including the top score in the study (−13.43 kcal/mol on HER2). SI yields larger per-molecule refinement gains (mean 0.82 – 1.18 kcal/mol across targets) but weaker population means overall, consistent with a headroom-limited regime. Rank-correlation analysis confirms that refinement gains depend strongly on seed quality. Beyond docking, the three strategies occupy distinct positions along an exploration-exploitation continuum: LATS favors synthetically tractable regions, SI maximizes internal diversity, and the Hybrid approach consistently balances novelty, diversity, and predicted binding strength. These results demonstrate that exploration and refinement function as complementary agentic design patterns, and that the ceiling on refinement gain is set by seed distribution quality, a finding with implications for agentic AI system design beyond molecular discovery.