This comprehensive benchmark evaluates language models' ability to effectively utilize tools and functions in complex scenarios.
We evaluate language models' ability to effectively use tools in single and multi-turn conversations. Our evaluation focuses on both basic functionality and edge cases that challenge real-world applicability.
| Category | Finding |
|---|---|
| Performance Champion | Gemini-2.0-flash dominates with 0.935 score at just $0.075 per million tokens, excelling in both complex tasks (0.95) and safety features (0.98) |
| Price-Performance Paradox | Top 3 models span 20x price difference yet only 3% performance gap, challenging pricing assumptions |
| Open Vs Closed Source | The new Mistral-small leads in open source models and performs similar to GPT-4o-mini at 0.83, signaling OSS maturity in tool calling |
| Reasoning Models | Although being great for reasoning, o1 and o3-mini are far from perfect scoring 0.87 and 0.84 respectively. DeepSeek V3 and R1 were excluded from rankings due to limited function support |
| Tool Miss Detection | Dataset averages of 0.59 and 0.78 reveal fundamental challenges in handling edge cases and maintaining context, even as models excel at basic tasks |
| Architecture Trade-offs | Long context vs parallel execution shows architectural limits: O1 leads context (0.98) but fails parallel tasks (0.43), while GPT-4o shows opposite pattern |
| Area | Recommendation |
|---|---|
| Task Complexity | Simple tasks work with most models. Complex workflows requiring multiple tools need models with 0.85+ scores in composite tests |
| Error Handling | Models with low tool selection scores need guardrails. Add validation layers and structured error recovery, especially for parameter collection |
| Context Management | Long conversations require either models strong in context retention or external context storage systems |
| Reasoning Models | While o1 and o3-mini excelled in function calling, DeepSeek V3 and R1 were excluded from rankings due to limited function support |
| Safety Controls | Add strict tool access controls for models weak in irrelevance detection. Include validation layers for inconsistent performers |
| Open Vs Closed Source | Private models lead in complex tasks, but open-source options work well for basic operations. Choose based on your scaling needs |
| Type | Samples | Category | Dataset Name | Purpose |
|---|---|---|---|---|
| Single-Turn | 100 + 100 | Single Function Call | xlam_single_tool_single_call | Evaluates basic ability to read documentation and make single function calls |
| 200 + 50 | Multiple Function Call | xlam_multiple_tool_multiple_call, xlam_single_tool_multiple_call | Tests parallel execution and result aggregation capabilities | |
| 100 | Irrelevant Query | BFCL_v3_irrelevance | Tests ability to recognize when available tools don't match user needs | |
| 100 | Long Context | tau_long_context | Assesses handling of extended interactions and complex instructions | |
| Multi-Turn | 50 + 30 | Single Function Call | BFCL_v3_multi_turn_base_single_func_call, toolscs_single_func_call | Tests basic conversational function calling abilities |
| 50 | Multiple Function Call | BFCL_v3_multi_turn_base_multi_func_call | Evaluates handling of multiple function calls in conversation | |
| 100 | Missing Function | BFCL_v3_multi_turn_miss_func | Tests graceful handling of unavailable tools | |
| 100 | Missing Parameters | BFCL_v3_multi_turn_miss_param | Assesses parameter collection and handling incomplete information | |
| 100 | Composite | BFCL_v3_multi_turn_composite | Tests overall robustness in complex scenarios |