Mapping biological mechanisms in cellular systems is a fundamental step in early stage drug discovery that serves to generate hypotheses on what disease-relevant molecular targets may effectively be modulated by pharmacological interventions. With the advent of high-throughput methods for measuring single-cell gene expression under genetic perturbations, we now have effective means for generating evidence for causal gene-gene interactions at scale. However, inferring graphical networks of the size typically encountered in real-world gene-gene interaction networks is difficult in terms of both achieving and evaluating faithfulness to the true underlying causal graph. Moreover, standardised benchmarks for comparing methods for causal discovery in perturbational single-cell data do not yet exist. Here, we introduce CausalBench - a comprehensive benchmark suite for evaluating network inference methods on large-scale perturbational single-cell gene expression data. CausalBench introduces several biologically meaningful performance metrics and operates on two large, curated and openly available benchmark data sets for evaluating methods on the inference of gene regulatory networks from single-cell data generated under perturbations. With real-world datasets consisting of over 200 000 training samples under interventions, CausalBench could potentially help facilitate advances in causal network inference by providing what is - to the best of our knowledge - the largest openly available test bed for causal discovery from real-world perturbation data to date.
Learn more about the CausalBench challenge for graph inference on perturbational gene expression data here.
- RPE1 day 7 Perturb-seq (RD7): targeting DepMap essential genes at day 7 after transduction
- K562 day 6 Perturb-seq (KD7): targeting DepMap essential genes at day 6 after transduction
- Observational: only observational data is given as training data to the model.
- Observational and partial interventional: observational as well as interventional data for part of the variables is given as training data to the model.
- Observational and full interventional: observational as well as interventional data for all the variables is given as training data to the model.
pip install causalbenchExample of command to run a model on the k562 dataset in the observational regime.
causalbench_run \
--dataset_name weissmann_k562 \
--output_directory /path/to/output/ \
--data_directory /path/to/data/storage \
--training_regime observational \
--model_name pc \
--subset_data 1.0 \
--model_seed 0 \
--do_filter \
--max_path_length -1 \
--omission_estimation_size 500Results are written to the folder at /path/to/output/, and processed datasets will be cached at /path/to/data/storage. See the MainApp class for more hyperparameter options, especially in the (partial) interventional setting.
New models can be easily added. The only contract for a model is to implement the [AbstractInferenceModel] class.
from causalscbench.models.abstract_model import AbstractInferenceModel
class FullyConnected(AbstractInferenceModel):
def __init__(self) -> None:
super().__init__()
def __call__(
self,
expression_matrix: np.array,
interventions: List[str],
gene_names: List[str],
training_regime: TrainingRegime,
seed: int = 0,
) -> List[Tuple]:
random.seed(seed)
edges = set()
for i in range(len(gene_names)):
a = gene_names[i]
for j in range(i + 1, len(gene_names)):
b = gene_names[j]
edges.add((a, b))
edges.add((b, a))
return list(edges)We welcome external contributions to help expand and improve the CausalBench project! If you're interested in contributing, please feel free to submit a PR. We recognize and appreciate all contributions by acknowledging contributors directly in the README under a dedicated section. If your contribution is associated with a published paper, we are also happy to cite it alongside your name.
Whether you’re proposing a new method, adding new features, improving existing methods, suggesting additional datasets, or enhancing/extending the evaluation metrics, your insights and expertise can significantly impact gene network inference research. Let's work together to advance our understanding and methodologies in causal inference within single-cell gene expression data!
Please consider citing, if you reference or use our methodology, code or results in your work:
@article{chevalley2025causalbench,
author = {Mathieu Chevalley and Yusuf H. Roohani and Arash Mehrjou and Jure Leskovec and Patrick Schwab},
title = {A large-scale benchmark for network inference from single-cell perturbation data},
journal = {Communications Biology},
year = {2025},
volume = {8},
number = {1},
pages = {412},
doi = {10.1038/s42003-025-07764-y},
url = {https://doi.org/10.1038/s42003-025-07764-y},
issn = {2399-3642},
}
Data in the /data_access/data folder were aggregated from the following resource: "ChIP-Atlas © Shinya Oki (Kyoto University) licensed under CC Attribution-Share Alike 4.0 International" link to license. The adapted datasets based on a part of the original database is here redistributed under the same license, which can be found in the LICENSE.txt file.
This codebase also links to muliple data sources to be downloaded. The associated licenses are summarized here:
Replogle et al (perturb-seq screen): https://gwps.wi.mit.edu/ (LICENSE: CC-BY-4.0)
CORUM: http://mips.helmholtz-muenchen.de/corum/ (LICENSE; CC-BY-NC)
StringDB : https://string-db.org/cgi/download.pl (LICENSE: CC-BY-4)
CellTalkDB: http://tcm.zju.edu.cn/celltalkdb/download.php (LICENSE: GNU General Public License v3.0)
Mathieu Chevalley, GSK plc
Yusuf H Roohani, GSK plc and Stanford University
Arash Mehrjou, GSK plc
Jure Leskovec, Stanford University
Patrick Schwab, GSK plc
MC, YR, AM and PS are employees and shareholders of GlaxoSmithKline plc.