Skip to content

[BFCL] Relocate Formatting Instructions and Function Documentation to System Prompt #593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Aug 25, 2024

Conversation

HuanzhiMao
Copy link
Collaborator

@HuanzhiMao HuanzhiMao commented Aug 22, 2024

Previously, formatting instructions and function documentation were included in the user prompt when interacting with models in prompting mode. However, these details are better suited for the system prompt, where they can more effectively guide the model's behaviour. This PR updates the model prompting process by moving the formatting instructions and function documentation to the system prompt, ensuring they are appropriately positioned for optimal model performance.

This will affect the leaderboard score.


Also in this PR:

  1. Update the model handlers to record the processed prompt/message and tools (if FC mode) in the result file when inference. This helps to identify if there are any issues with the pre-processing phase.
  2. Fix 6 dataset issues: irrelevance_49, live_irrelevance_157-18-1, live_simple_79-40-0, live_parallel_0-0-0, live_parallel_4-1-0, live_parallel_5-2-0

@HuanzhiMao HuanzhiMao marked this pull request as ready for review August 23, 2024 05:55
Copy link
Collaborator

@CharlieJCJ CharlieJCJ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on Gorilla openfunctions and GPT models, both prompt and FC models, LGTM.

Logging of processed_message will add a lot of debuggability for adding new model handlers.

@ShishirPatil ShishirPatil merged commit 3850c2b into ShishirPatil:main Aug 25, 2024
@HuanzhiMao HuanzhiMao added the BFCL-General General BFCL Issue label Aug 25, 2024
@HuanzhiMao HuanzhiMao deleted the message_format branch August 25, 2024 06:25
ShishirPatil pushed a commit that referenced this pull request Aug 29, 2024
This PR updates the leaderboard to reflect the change in score due to
#593 .

---------

Co-authored-by: Charlie Cheng-Jie Ji <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BFCL-General General BFCL Issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants