Support for hyperparameter/configuration libraries

[ncilfone]

👋

I think reproducibility is something that we are still lacking within the ML community so I really like the idea and what you are trying to accomplish with replicate!

First thought that popped into my head is adding a more structured and less bolierplate-y way to track and replicate model hyperparameters and data configurations.

For instance from the tutorial:

# Create an "experiment". This represents a run of your training script.
# It saves the training code at the given path and any hyperparameters.
experiment = replicate.init(
    path=".",
    params={"learning_rate": learning_rate, "num_epochs": num_epochs},
)

...

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--learning_rate", type=float, default=0.01)
    parser.add_argument("--num_epochs", type=int, default=100)
    args = parser.parse_args()
    train(args.learning_rate, args.num_epochs)

This relies on two things that I would think you want to try and combat against... (1) relying on the user to make sure that all the hyperparameters that are needed/wanted for replication are defined in the init call and (2) writing a ton of argparser boilerplate so that the hyperparameters can be reused in replicate runs (this is my guess of what's going on under the hood without looking). I think you can tackle, or at least partially tackle, these with one of the existing open source hyperparameter/configuration libraries.

Slightly biased towards a library that we wrote Spock that handles complex parameter configurations, reproducibility, type checking, complex parameter types all with minimal boilerplate and automatic reading of parameter definitions from conf files (yaml, toml, json). I think something like this could help handle the configuration side of replicate in a more structured and reproducible way, especially since the entire configuration can be dumped to a valid/useable config file with a simple chained .save() call which I think aligns with the philosophy that 'It's just plain old files on S3'.

(To be unbiased there is also OmegaConf that has 'Structured Configs' that do something very similar, but is a little less built out on the reproducibility and complex types imo)

Anyways, happy to help brainstorm or discuss more and contribute if possible!

Nick

[bfirsh]

This is a great idea. We're quite intentionally not doing any deep integration with configuration libraries. The idea is that Replicate just saves files and dictionaries, so whatever way you're already tracking your configuration, you can just stuff that into one of those things.

You're right using argparse in the example is verbose and not ideal, but it's kinda intentionally supposed to be simple and not opinionated. I wonder if there's a better solution that we can do with the Python standard library?

Something I think that would be great is some lightweight documentation showing how to integrate Replicate with configuration libraries. I've created an issue to track that. If you wanted to help out with that, that would be amazing!

I think we're also interested in making training completely reproducible. The problem with the current system of just recording the params, is we don't know how to feed those params back into experiments to reproduce the training. This might mean deeper integration with configuration libraries, or maybe doing some lightweight configuration management ourselves.

Either way -- would be great to brainstorm some of this stuff with you too. We're in Discord if you want to chat!