-
Notifications
You must be signed in to change notification settings - Fork 1.2k
[BFCL] Add the option to log to WandB during bfcl evaluate #736
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BFCL] Add the option to log to WandB during bfcl evaluate #736
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR @ThomasRochefortB !
Two suggestions:
- We should add
wandb
to thepyproject.toml
; otherwise the pipeline will break. Maybe as an optional-dependency sincewandb
is not required for evaluation to complete? - Maybe we should add the wandb_project to an environment variable (in the
.env
file) and just use a CLI flag to signal that the result should be added to the wandb because it's likely user would use the same wandb_project across runs. So they don't need to type it in the command line everytime, and it makes the code cleaner.
What do you think?
Thank you for the feedback @HuanzhiMao ! I think these are 2 great suggestions. I will adjust my branch accordingly. |
Sounds good! |
There you go @HuanzhiMao ! I have made the following changes:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Thanks for the PR @ThomasRochefortB !
In our internal use of BFCL at Valence Labs, we use WandB extensively to centralize the results of our benchmarking.
This PR adds a
--wandb-project
CLI argument to thebfcl evaluate
command to upload the generated.csv
files to wandb. I wanted to open this PR to see if this would be an feature of interest for BFCL :)Here is an example command:
This will log the
data_live.csv
,data_non_live.csv
anddata_overall.csv
as dataframe artifacts on the WandBwandb_project
under thewandb_entity
.