Evaluation design shapes estimates of cross-donor generalization in single-cell classification of cell-type

Abstract

Motivation: Single-cell cell-type classifiers are often evaluated using random cell-level splits, although cells from the same donor are not independent. We asked how evaluation design affects estimates of cross-donor generalization. Results: Across PBMC and lung datasets, random cell-level splitting consistently yielded higher macro F1 than donor-held-out evaluation across multiple expression representations. This gap persisted after batch covariate adjustment and donor-level mixed-effects modeling. In contrast, evaluation gaps were minimal in near-ceiling kidney, pancreas, and blood benchmarks and in a randomized-label floor control. These results show that evaluation bias depends on the underlying classification regime and that donor-held-out evaluation is necessary when claims concern generalization to unseen individuals. Availability and Implementation: Code and analyses are available on GitHub at https://github.com/danielchen05/scRNA-cross-donor-generalization/ and archived at Zenodo at 10.5281/zenodo.23669184. All data are publicly available and cited.