FCVOpt is a Python package for "Fractional Cross-Validation"in hyperparameter optimization. It implements efficient hyperparameter tuning by evaluating only a fraction of cross-validation folds using hierarchical Gaussian processes.

The documentation is available at https://syerramilli.github.io/fcvopt/.

🚀 Key Features:

• Efficient Optimization: Evaluate hyperparameters using only a subset of CV folds
• Hierarchical Gaussian Processes: Model fold-wise correlations for better predictions
• MLflow Integration: Automatic experiment tracking and model versioning
• Framework Support: Scikit-learn, XGBoost, PyTorch (via Skorch), and more

git clone https://github.com/syerramilli/fcvopt.git
cd fcvopt
pip install .

With optional dependencies:

pip install .[experiments]  # For reproducing the results from the paper
pip install .[docs]         # For building documentation

If you use this code in your research, please cite the following paper:

@article{yerramilli2025fractional,
    author = {Suraj Yerramilli and Daniel W. Apley},
    title = {Fractional Cross-Validation for Optimizing Hyperparameters of Supervised Learning Algorithms},
    journal = {Technometrics},
    year = {2025},
    doi = {10.1080/00401706.2025.2515926},
}

The experiments are all contained in the experiments folder. Each subdirectory within this folder contains scripts files to run each case study in the paper. Refer to the README file within each of the subdirectories for instructions to run the files.

For reproducibility, we provide two options for setting up the environment to run the experiments: a virtual environment using venv and a Docker container.

The bash script file venv_setup.sh can be used to create a virtual environment and install the required packages. Ensure you have Python >= 3.8 and <=3.12 installed.

To run the script, use the following commands:

chmod +x venv_setup.sh
./venv_setup.sh

Note: The experiments involving the SMAC algorithm require the smac library, which in turn requires the building and compliling the pyrfr package While the main functions of fcvopt do not depend on pyrfr, you might encounter build issues during its installation if you do not have a C++ compiler and the swig binary installed on your system.

The Dockerfile is provided to run the experiments in a container with the fcvopt package and all the required dependencies. The Dockerfile is based on the Python 3.10 debian image. To build the image, run the following command:

docker build -t fcvopt_test .

To run the container with the files in the experiments folder mounted, run the following command:

docker run -v <path_to_experiments_folder>:/app/experiments -it fcvopt_test

This will launch the container and open a bash shell. The experiments directory will be mounted in the container at /app/experiments. Mounting the directory allows you to access the files in the experiments folder from within the container, and any changes made to the files will be reflected in your local directory. Replace <path_to_experiments_folder> with the absolute path to your local experiments directory. Relative paths will not work, as the container will not have access to your local file system. On Linux and MacOS, you can use the $(pwd) command to get the absolute path of the current directory. For example:

docker run -v $(pwd)/experiments:/app/experiments -it fcvopt_test

Once inside the container, you can navigate to the /app/experiments directory and run the experiments as needed. For example:

cd experiments
bash reproduce_rf.sh

Note: On Ubuntu/Debian, you may need administrative privileges to run the Docker commands. You can do this by adding sudo before the command.