Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 12.1k

Running on CPU Upgrade

Hi @csabakecskemeti ,

Not a lame question at all! Let me explain how the task group score (like MUSR) is calculated with a small code example.

The score for a task group is the average of the normalized scores for all tasks within that group. For your example:

murder_mysteries = 0.5280
object_placements = 0.3008
team_allocation=0.2200

The MUSR score is calculated as:

scores = [0.5280, 0.3008, 0.2200]
musr_score = sum(scores) / len(scores)
print(f"MUSR Score: {musr_score:.4f}")
# Output: MUSR Score: 0.3496

In our parser, this simplified logic might look like this:

import numpy as np

# Example scores for tasks within the group
task_scores = {
    "leaderboard_musr_murder_mysteries": 0.5280,
    "leaderboard_musr_object_placements": 0.3008,
    "leaderboard_musr_team_allocation": 0.2200,
}

# Compute the average score for the group
task_group_name = "MUSR"
task_group_score = np.mean(list(task_scores.values()))

print(f"{task_group_name} Score: {task_group_score:.4f}")
# Output: MUSR Score: 0.3496

We normalize the scores to ensure consistency across tasks, especially when different tasks have varying metrics or scales. This guarantees a fair representation in the group score. You will find more info in our documentation

I hope this clears things up. Feel free to ask if you have more questions!

Thanks

I'm closing this discussion, feel free to open a new one in case of any questions!