Add CALM models

[jgreer013]

Add the following new models to the leaderboard:

• uiuc-convai/CALM-8B
• uiuc-convai/CALM-70B
• uiuc-convai/CALM-405B

[HuanzhiMao]

Thanks for the PR and welcome @jgreer013 !

I wanted to mention that due to our current compute constraints (we only have 8 H100 GPUs available), we’re unable to benchmark any 405B models at the moment—unless they’re hosted externally and accessible via API calls.

[jgreer013]

@HuanzhiMao Assuming we can get it hosted, what would the process be to give you access for submission?

Alternatively, would an I8/FP8 405B work? That should fit within the 640GB of VRAM.

[jgreer013]

Alternatively, we could share the 405B generate outputs directly and you all can validate the score.

[HuanzhiMao]

Replied over email. Copied over here for the record.

Hosting the model would indeed be ideal. Functionary has done this before by spinning up an OpenAI-compatible server and sharing the endpoint via DM on Discord. Once I have that endpoint, I can simply update the base_url in our handler to direct inference there. For reference, it would look something like this:

base_url="https://xxxx-8000.proxy.runpod.net/v1"
model="meetkai/functionary-medium-v3.1"

Regarding the I8/FP8 405B option, that should work as well—though we would note this on the leaderboard as uiuc-convai/CALM-405B (FP8) to reflect the difference in precision.

As for sharing pre-generated outputs, we unfortunately can’t accept them because we need to ensure fairness and consistency in our evaluations. There’s no way for us to verify that the outputs haven’t been altered.