Skip to content

Commit 849a815

Browse files
authored
fix tokenizer of chatglm2 (#2711)
1 parent 8bd422b commit 849a815

File tree

1 file changed

+4
-1
lines changed

1 file changed

+4
-1
lines changed

fastchat/train/train.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,9 @@
3636
@dataclass
3737
class ModelArguments:
3838
model_name_or_path: Optional[str] = field(default="facebook/opt-125m")
39+
padding_side: str = field(
40+
default="right", metadata={"help": "The padding side in tokenizer"}
41+
)
3942

4043

4144
@dataclass
@@ -274,7 +277,7 @@ def train():
274277
model_args.model_name_or_path,
275278
cache_dir=training_args.cache_dir,
276279
model_max_length=training_args.model_max_length,
277-
padding_side="right",
280+
padding_side=model_args.padding_side,
278281
use_fast=False,
279282
)
280283
tokenizer.pad_token = tokenizer.unk_token

0 commit comments

Comments
 (0)