inference/huggingface/text-generation example dies with: ImportError: cannot import name 'LlamaTokenizerFast' from 'transformers'

[jagane-infinstor]

Looks like we need a newer version of the transformers library. Here's the full error:

$ deepspeed --num_gpus 1 inference-test.py --name bigscience/bloom-3b --batch_size 2
[2023-07-17 19:59:46,537] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-17 19:59:46,870] [WARNING] [runner.py:196:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2023-07-17 19:59:48,675] [INFO] [runner.py:555:main] cmd = /opt/conda/envs/deepspeed1/bin/python3.10 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None inference-test.py --name bigscience/bloom-3b --batch_size 2
[2023-07-17 19:59:50,604] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-17 19:59:50,934] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [0]}
[2023-07-17 19:59:50,935] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=1, node_rank=0
[2023-07-17 19:59:50,935] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
[2023-07-17 19:59:50,935] [INFO] [launch.py:163:main] dist_world_size=1
[2023-07-17 19:59:50,935] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0
[2023-07-17 19:59:53,004] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):
File "/home/jagane/working/DeepSpeedExamples/inference/huggingface/text-generation/inference-test.py", line 8, in
from utils import DSPipeline
File "/home/jagane/working/DeepSpeedExamples/inference/huggingface/text-generation/utils.py", line 12, in
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer, LlamaTokenizerFast
ImportError: cannot import name 'LlamaTokenizerFast' from 'transformers' (/opt/conda/envs/deepspeed1/lib/python3.10/site-packages/transformers/init.py)
[2023-07-17 19:59:53,939] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 27100
[2023-07-17 19:59:53,939] [ERROR] [launch.py:321:sigkill_handler] ['/opt/conda/envs/deepspeed1/bin/python3.10', '-u', 'inference-test.py', '--local_rank=0', '--name', 'bigscience/bloom-3b', '--batch_size', '2'] exits with return code = 1