Chat with lmsys/fastchat-t5-3b-v1.0

[llmwesee]

Whenever i try to query through PDF using lmsys/fastchat-t5-3b-v1.0 then it always gave sources as a answer. I tried different different prompts according to the https://github.com/h2oai/h2ogpt/blob/main/docs/FAQ.md#adding-models but noting works for me.
For more reference I attached the screenshot.
Kindly help me to resolve this.

[pseudotensor]

lm-sys/FastChat#1264 (comment)

Looks like closest we have built-in is "guanaco" or "instruct_vicuna". Try those.

[llmwesee]

Not working at all. when trying "instruct_vicuna" the following runtime error popup:
if torch.all((stop == input_ids[0][-len(stop):])).item(): RuntimeError: The size of tensor a (6) must match the size of tensor b (2) at non-singleton dimension 0

And when trying "guanaco" it only gives sources information, not creating any context.

[pseudotensor]

It is not clear from the docs on fastsys what prompting should be

[llmwesee]

Then how someone can do prompting? when we trying it on demo provided by fastsys it works good but when trying it here then it slows and not working at all.
There is any other method we should try.

[pseudotensor]

One can do custom prompt, but I can see what they intend.

[llmwesee]

ok I'll wait for it.

[pseudotensor]

@llmwesee Can try now if using source code. Windows installer built too

[llmwesee]

Now, what prompt_type should I finally use for better generation?

[pseudotensor]

It defaults to one_shot automatically, but you can choose instruct_vicuna too. instruct_vicuna is leaner, but may give less quality output (but will give you more room for context like documents).

Both are tested: https://github.com/h2oai/h2ogpt/pull/690/files#diff-a40001c71f9d4fa20490fda2b0fe0cbfa9ddcec119a36ba41bf1ff6e90d7f5e1R1284