Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BFCL] Retire Executable Categories from Leaderboard #943

Merged
merged 14 commits into from
Apr 10, 2025

Conversation

HuanzhiMao
Copy link
Collaborator

BFCL V1 introduced several “executable” categories (rest, exec_simple, exec_multiple, exec_parallel, exec_parallel_multiple) that rely on external APIs. However, these categories have drawbacks that frequently arise in community feedback:

  1. Unreliable API availability: The external API can fail randomly, resulting in frequent sanity-check failures and inconsistent evaluation results.
  2. Integration difficulties: Foundation model providers struggle to incorporate executable categories into their training pipelines due to the complexity involved.
  3. API key management: While the APIs offer free plans, these often lack sufficient bandwidth for evaluating multiple models together, forcing users to upgrade to paid plans.
  4. Slow performance: External API calls add latency, slowing down the evaluation process.
  5. AST sufficiency: AST-based checks already serve as a robust proxy for executability, reducing the need for external APIs.

Because of these issues, this PR removes all executable categories from BFCL and excludes them from the overall leaderboard accuracy calculation.

@HuanzhiMao HuanzhiMao added BFCL-General General BFCL Issue BFCL-Dataset BFCL Dataset-Related Issue labels Mar 16, 2025
Copy link
Collaborator

@Fanjia-Yan Fanjia-Yan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HuanzhiMao HuanzhiMao merged commit 4ed459c into ShishirPatil:main Apr 10, 2025
@HuanzhiMao HuanzhiMao deleted the exec_retire branch April 10, 2025 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BFCL-Dataset BFCL Dataset-Related Issue BFCL-General General BFCL Issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants