MoE model run failed

#514
by vihangd - opened

Hello,

I am trying to understand why my recent submission failed (vihangd/smartsolmix-4x10.7b-v1). I tested its evaluation locally with a limited number of examples.

https://huggingface.co./datasets/open-llm-leaderboard/requests/blob/main/vihangd/smartsolmix-4x10.7b-v1_eval_request_False_float16_Original.json

Could you please help me shed some light on this situation? Any insights you could provide would be greatly appreciated.

Regards,
V

Open LLM Leaderboard org

Hi!
Thanks a lot for the complete issue!

Just checked the logs and it actually did not fail, your results are here. I manually updated your request file so the results get pulled to the leaderboard at the next rebuild but we will need to investigate why it appeared as failed.

Hmm that's even more strange, but thanks a lot for investigating.

Open LLM Leaderboard org

Since you are the second user to which it happened, I suspect we have a small bug in our backend (probably a file check which fails). Are you OK with me closing this issue?

No worries, closing it for now. Thanks a lot again. Oh and have a great new year.

Regards,
V

vihangd changed discussion status to closed
Open LLM Leaderboard org

You too!! :D

Sign up or log in to comment