Skip to content

Conversation

ThomasRochefortB
Copy link
Contributor

@ThomasRochefortB ThomasRochefortB commented Nov 4, 2024

In our internal use of BFCL at Valence Labs, we use WandB extensively to centralize the results of our benchmarking.
This PR adds a --wandb-project CLI argument to the bfcl evaluate command to upload the generated .csv files to wandb. I wanted to open this PR to see if this would be an feature of interest for BFCL :)

Here is an example command:

bfcl evaluate --model gpt-3.5-turbo-0125  
--wandb-project <wandb_entity:wandb_project>

This will log the data_live.csv, data_non_live.csv and data_overall.csv as dataframe artifacts on the WandB wandb_project under the wandb_entity.

Copy link
Collaborator

@HuanzhiMao HuanzhiMao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @ThomasRochefortB !

Two suggestions:

  1. We should add wandb to the pyproject.toml; otherwise the pipeline will break. Maybe as an optional-dependency since wandb is not required for evaluation to complete?
  2. Maybe we should add the wandb_project to an environment variable (in the .env file) and just use a CLI flag to signal that the result should be added to the wandb because it's likely user would use the same wandb_project across runs. So they don't need to type it in the command line everytime, and it makes the code cleaner.

What do you think?

@ThomasRochefortB
Copy link
Contributor Author

Thank you for the feedback @HuanzhiMao !

I think these are 2 great suggestions. I will adjust my branch accordingly.

@HuanzhiMao
Copy link
Collaborator

Thank you for the feedback @HuanzhiMao !

I think these are 2 great suggestions. I will adjust my branch accordingly.

Sounds good!

@ThomasRochefortB ThomasRochefortB changed the title Add the option to log to WandB during bfcl evaluate [BFCL] Add the option to log to WandB during bfcl evaluate Nov 11, 2024
@ThomasRochefortB
Copy link
Contributor Author

There you go @HuanzhiMao ! I have made the following changes:

  • Added an optional env var WANDB_BFCL_PROJECT in the .env.example
  • Modified the init_wandb function to read from the env var
  • Added Wandb as an optional dependencies on the pyoroject.toml

Copy link
Collaborator

@HuanzhiMao HuanzhiMao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Thanks for the PR @ThomasRochefortB !

@HuanzhiMao HuanzhiMao merged commit 0fc2bd2 into ShishirPatil:main Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants