pyinstaller has some bug that results in improper packaging of tiktoken #43
Description
What could be the fix for this error. I am trying out the library for the first time.
import tiktoken
enc = tiktoken.get_encoding("gpt2")
assert enc.decode(enc.encode("hello world")) == "hello world"
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Input In [47], in <cell line: 2>()
1 import tiktoken
----> 2 enc = tiktoken.get_encoding("gpt2")
3 assert enc.decode(enc.encode("hello world")) == "hello world"
File ~/work/p3ds/lib/python3.10/site-packages/tiktoken/registry.py:60, in get_encoding(encoding_name)
57 assert ENCODING_CONSTRUCTORS is not None
59 if encoding_name not in ENCODING_CONSTRUCTORS:
---> 60 raise ValueError(f"Unknown encoding {encoding_name}")
62 constructor = ENCODING_CONSTRUCTORS[encoding_name]
63 enc = Encoding(**constructor())
ValueError: Unknown encoding gpt2