Skip to content

Commit b9de8b0

Browse files
authored
[Update] Unset disallowed_special token for Openai model (#1960)
1 parent c985992 commit b9de8b0

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

opencompass/models/openai_api.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -399,7 +399,7 @@ def get_token_len(self, prompt: str) -> int:
399399
self.logger.info(
400400
f'Successfully load default tiktoken tokenizer: '
401401
f' {default_tokenizer}')
402-
return len(enc.encode(prompt))
402+
return len(enc.encode(prompt, disallowed_special=()))
403403

404404
def _bin_trim(self, prompt: str, num_token: int, mode: str) -> str:
405405
"""Get a suffix of prompt which is no longer than num_token tokens.

0 commit comments

Comments
 (0)