Code for the paper Position: All Current Generative Fidelity and Diversity Metrics are Flawed, accepted to ICML 2025.
We use Poetry to manage dependencies. Install Poetry according to their instructions, and use
poetry install
to install all our dependencies.
We use Snakemake to run our code. The command
poetry run snakemake -j 5
runs all sanity checks and compiles their results (Tables 3 and 4 in the paper).
This uses 5 processes in parallel, controlled by the -j 5 argument.
Notebooks that plot all of our figures are in the plotting directory.
The following code is taken from external sources, with minor modifications. Note that these are licensed under their original licenses, as specified below.
- Precision, recall, density, coverage in
metrics/prdc.py: https://github.com/clovaai/generative-evaluation-prdc, MIT license - Alpha-precision, beta-recall in
metrics/alpha_precision_beta_recall: https://github.com/ahmedmalaa/evaluating-generative-models, MIT license or BSD 3-clause license (repo has MIT, files say BSD 3-clause) - precision and recall cover in
metrics/precision_recall_cover: https://github.com/FasilCheema/GenerativeMetrics, GPL-3.0 license - Symmetric precision and recall in
metrics/sym_precision_recall: https://github.com/mahyarkoy/emergent_asymmetry_pr, MIT license - Probabilistic precision and recall in
metrics/pp_pr.py: https://github.com/kdst-team/Probablistic_precision_recall, MIT license