Support for multiple LoRA

[elevran]

Need to support mapping of LoRA to base model.
Client uses LoRA name in request and this needs to be mapped to the relevant base model (and InferencePool).
With the removal of InferenceModel from the IGW API, there is no defined way to map a LoRA name to the model.
This can be learnt from convention (e.g., use base#lora, or similar, as naming scheme), configuration (e.g., CRDs, should probably consider a design along the lines of EndpointSlice to improve scale) or the model servers directly (e.g., /v1/models).
This could be done/consumed by BBR, EPP or both.

• provide working examples based on available functionality
• explore options for implementation and draft community proposal

[elevran]

Per discussion on inference-scheduler community meeting 10-Spe (notes), for small-mid scale deployments, the HTTPRoute match rules can be used to assign different LoRA model values to an inference pool.
There is a limit of 64 items in the match array. This means that BBR can extract the model value from body and set it to the header. As long as the header values (the LoRA names) are known a-priori and <64, this scheme should work.
UX and scale (beyond 64 LoRA) can be improved in future iterations (e.g., BBR configurations, integration with LoRA loading and distribution, etc.).
@kfswain @nilig

[ahg-g]

I thought we could group all LoRAs that target a specific InfPool into one match (using an in type selection), but that doesn't seem to be supported unfortunately, it is an exact match of one only, and the HTTPHeaderMatch array is ANDed.