Enable thinking models

[rlouf]

A v0 would be allowing users to instantiate the logits processor ThinkingLogitsProcessor with the end of thinking token and pass it to the generator.

model = ...
logits_processor = ThinkingLogitsProcessor("</think>", output_type)
generator = Generator(model, processor=logits_processor)

The implementation is simple:

1. While </think> has not been observed, __call__ is a pass-through.
2. After <think> has been generated, mask the logits to generate the type passed by the user.

[torchss]

This is fantastic. Would this also have the ability to put a constraint on how much "thinking" the model does and force thinking to stop based off an user provided parameter?

[TheooJ]

Agree that this would be very useful. For now, the best I've found is to add reasoning: str to the top of the type, which significantly improves performance (both for thinking and non-thinking models).

Could there also be a start_processing_logits=</think> directly in the Generator ? In any case I like the proposed API too.

[RobinPicard]

Adding reasoning: str to the top of the type is when you're using a Pydantic class to represent a JSON schema, right?

[rlouf]

A quick note, unrelated to the discussion. During internal evaluations, we've come across a couple of instances where the model "thinks" beyond the total context length. This makes me think we could add an option to limit the number of thinking tokens.

[torchss]

This makes me think we could add an option to limit the number of thinking tokens

This WOULD BE PHENOMENAL! A lot of my dsPy metrics fail because the LLM exhausted its tokens. I would assume a potential fix would be another custom logits processor.

Really looking forward to it as this would be yet another reason to integrate outlines into dsPy (related: #1632 (reply in thread))