Logprobs and loglikelihood #72
Conversation
| _ **delta** *(object or null)*: An optional field that can contain additional metadata about the message, typically `null`. | ||
| _ **finish_reason** _(string)_: The reason why the message generation was stopped, such as reaching the maximum length (`"length"`). | ||
| _ **finish_reason** _(string)_: The reason why the message generation was stopped, such as reaching the maximum length (`"length"`). | ||
| _ **logprobs** _(object)_: An object representing the token, its log probability and the most probable tokens to this one. |
There was a problem hiding this comment.
"the most probable tokens to this one" what does this mean?
There was a problem hiding this comment.
@Red-Caesar - Are you saying the other tokens most likely to be selected? I.e. if "cat" was chosen, but "dog" and "mouse" were the 2nd and 3rd most likely tokens to be selected, those would be included in the output?
There was a problem hiding this comment.
Yes, it will be included in the response if we set the top_logprobs > 1. For example, we ask: "Create a story about a cat".
The response will be in the following format:
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": " In a quaint, cobblestone town,",
"tool_calls": null
},
"logprobs": {
"content": [
{
"token": " In",
"logprob": -0.7101921439170837,
"bytes": null,
"top_logprobs": [
{
"token": " In",
"logprob": -0.7101921439170837,
"bytes": null
},
{
"token": " Once",
"logprob": -1.9485827684402466,
"bytes": null
}
]
},
.... other tokens
|
I'm a bit concerned about the loglikelihood because if users use the OpenAI API, they may have some difficulties sending this parameter correctly. It should be sent in a specific format: import openai
client = openai.OpenAI(
base_url=ENDPOINT + "/v1",
api_key=OCTOAI_TOKEN,
)
completion = client.chat.completions.create(
model="mistral-7b-instruct",
messages=[
{
"role": "user",
"content": "Create a story about a cat",
}
],
extra_body={"loglikelihood":True},
)So, should we write more about this? If so, I'm not sure where in the documentation it would be best to do so. |
3 participants
Update docs with logprobs and loglikelihood.
Additional notebook to run completions with these parameters.