Spaces:
Running
on
CPU Upgrade
Update leaderboard so can support newer models?
Any plans to update the leaderboard, so that it can support newer models such as mixtral (8x7B & 8x22B), gemma, & phi-3 ?
thx!
Hi @CombinHorizon , thanks for the question. Technically speaking, the leaderboard accepts all models supported by vLLM. We have already evaluated mixtral 8x7b and are busy running evals for other larger models (see https://github.com/logikon-ai/cot-eval/issues).
As to submitting larger models via the "submit" tab, the issue is really compute. It's just for lack of GPU resources that we're currently prioritising 7b models. But the plan is to get more compute. :-)
Brief follow-up: We've just upgraded to the latest lm-eval-harness and vllm releases, and successfully ran phi-3. Feel free to submit models based on latest architecture.