You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[BFCL Chore] Fix Functionary Medium 3.1 model name & add readme parallel inference (#577)
Changes:
- Fix Functionary Medium 3.1 model version name in
`eval_runner_helper.py`
- add readme parallel inference
---------
Co-authored-by: Huanzhi (Hans) Mao <[email protected]>
You can optionally specify the number of threads to use for *parallel inference* by setting the `--num-threads` flag to speed up inference for **hosted models**, not applicable for OSS models.
74
75
75
76
For available options for `MODEL_NAME` and `TEST_CATEGORY`, please refer to the [Models Available](#models-available) and [Available Test Category](#available-test-category) section below.
76
77
@@ -222,7 +223,7 @@ Some companies have proposed some optimization strategies in their models' handl
222
223
223
224
*[August 8, 2024][#574](https://github.com/ShishirPatil/gorilla/pull/574): Set temperature to 0.001 for all models for consistency and reproducibility.
224
225
*[August 7, 2024][#571](https://github.com/ShishirPatil/gorilla/pull/571): Support parallel inference for hosted models. User can specify the number of threads to use for parallel inference by setting the `--num-threads` flag. The default is 1, which means no parallel inference.
225
-
*[August 6, 2024][#569](https://github.com/ShishirPatil/gorilla/pull/569), [#570](https://github.com/ShishirPatil/gorilla/pull/570): Add the following new models to the leaderboard:
226
+
*[August 6, 2024][#569](https://github.com/ShishirPatil/gorilla/pull/569), [#570](https://github.com/ShishirPatil/gorilla/pull/570), [#573](https://github.com/ShishirPatil/gorilla/pull/573): Add the following new models to the leaderboard:
226
227
*`open-mistral-nemo-2407`
227
228
*`open-mistral-nemo-2407-FC-Any`
228
229
*`open-mistral-nemo-2407-FC-Auto`
@@ -234,6 +235,9 @@ Some companies have proposed some optimization strategies in their models' handl
234
235
*`gpt-4o-mini-2024-07-18-FC`
235
236
*`gpt-4o-2024-08-06`
236
237
*`gpt-4o-2024-08-06-FC`
238
+
*`meetkai/functionary-medium-v3.1-FC`
239
+
*`meetkai/functionary-small-v3.1-FC`
240
+
*`meetkai/functionary-small-v3.2-FC`
237
241
*[August 5, 2024][#568](https://github.com/ShishirPatil/gorilla/pull/568): Rephrase the question prompt for the `executable_parallel_function` category to remove potentially misleading information implying multi-turn function calls.
238
242
*[August 4, 2024][#557](https://github.com/ShishirPatil/gorilla/pull/557): Bug fix in the possible answers.
0 commit comments