Prevent crash if model-index metadata not specified in model card

#59
by tomaarsen HF staff - opened
Massive Text Embedding Benchmark org
edited Dec 30, 2023

Hello!

Pull Request overview

  • Prevent crash if model-index metadata not specified in model card

Details

First of all, apologies for the leaderboard being down earlier. I realised a bit too late that I couldn't safely restart the app without locally trying it first. Upon receiving the crash, I started running it locally, but it ran for a long while before everything was downloaded for the first time.
This PR prevents the crash by ignoring models without any model-index metadata, although I recognize that perhaps the crash is "on purpose" to identify models that should be skipped (i.e. added to MODELS_TO_SKIP). The ideal solution may be to raise a warning or store the "incorrect" models in a list so that a warning can be thrown for them at the end of the script. Feel free to let me know your thoughts!

  • Tom Aarsen
tomaarsen changed pull request status to open
Massive Text Embedding Benchmark org

Nice! Indeed the crash is partly on purpose to identify incorrect models but I think this is no longer OK as more & more models are submitted so I am okay with this change!

Muennighoff changed pull request status to merged

Sign up or log in to comment