Any updates on redesigning the leaderboard?

#595
by TNTOutburst - opened

I thought around December or so, there were a lot of conversations on ways to redesign how the leaderboard works. I was wondering if you have any updates on what you are working on or if you are still looking for solutions.

deleted
Open LLM Leaderboard org
edited Feb 12

Hi!
To list all the topics I remember:

  • a number of these conversations were about adding new leaderboards (for example, with rolling updates or private benchmarks), not necessarily changing the Open LLM Leaderboard itself > you can find some of the new leaderboards that were created with partners here.
  • wrt contamination, we've had to put these conversations on hold while we were focusing on the release of our mini evaluation suite, lighteval. We'll come back to them as soon as we can!
  • we've also increased the efforts on metadata (things like requesting licenses, adding flags for moerges, ...).

We're also looking to hire an intern, because we are stretched quite thin (but hiring takes a lot of time) - once this is done, we'll likely come back in full force on some of these topics :)

clefourrier changed discussion status to closed

Sign up or log in to comment