Supporting multiple OpenAI-compatible providers in one Api object

[shaltielshmid]

I couldn't figure out how to do it with LlmTornado - I want to add multiple OpenAI-compatible endpoint providers into the same object.

It seems to me like it won't work, since only one EndpointProvider is supported for each LlmProvider:

https://github.com/lofcz/LlmTornado/blob/master/src/LlmTornado/TornadoApi.cs#L152-L153

It would also need to be handled with regard to specifying the ChatModel - how to disambiguate which EndpointProvider each model goes to, when they all have the same LlmProvider.

It might just be a non-starter and I should use multiple TorandoApi objects, but I've encountered more than once the need to support both OpenAI, and other servers serving using an OpenAI-compatible endpoint.

What do you think?

[lofcz]

The api objects are fairly lightweight, since endpoints are lazy-wrapped. What's the benefit of having one instance in your use case?

As you've pointed out, providers are now represented via a dictionary, it could be a dictionary, where the value is a list, but there would need to be some clear benefits, should we make this change.

[shaltielshmid]

It's mostly for elegance.
The Api object already handles multiple authentications in an elegant way, so it was mostly to keep that complication behind the scenes. Definitely not crucial.

[lofcz]

This could be a fairly common use case, maybe we could ship an abstraction for that out of the box. The user would configure named TornadoApi instances and provide a function that routes the requested model to one of them. We could also support the lifetime configuration of the instances to enable singleton/scoped/transient behavior.