config.yaml: model.capabilities and autodetection

[owtaylor]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- Continue version: git main

Description

The handling of model.capabilities is: that the capabilities array is present, then it defines the capabilities and no autodetection is done - so if you have capabilities: ["tool_use"] that means that the model doesn't support image_input.

It's unclear from the docs whether this is intentional or not. There's a tradeoff:

• If you autodetect for missing capabilities, there is no way to disable a capability
• If you don't autodetect for missing capabilities, then existing config files don't get new autodetected capabilities

The code has it both ways:

  let options: LLMOptions = {
[...]
    capabilities: {
      tools: model.capabilities?.includes("tool_use"),
      uploadImage: model.capabilities?.includes("image_input"),
    },
  };

  // Model capabilities - need to be undefined if not found
  // To fallback to our autodetection
  if (capabilities?.find((c) => c === "tool_use")) {
    options.capabilities = {
      ...options.capabilities,
      tools: true,
    };
  }

  if (capabilities?.find((c) => c === "image_input")) {
    options.capabilities = {
      ...options.capabilities,
      uploadImage: true,
    };
  }

The first block which wins was added by @sestinj on 03-10, the second block by @RomneyDa on 03-04.

To reproduce

No response

Log output

[Patrick-Erichsen]

I'm relatively confident that it would be preferable to keep autodetect even when the user has explicitly defined a given capability.

The original reason we added this feature was for image_input to support long-tail models that we didn't have explicit configuration for. So, the original spirit of this work was to give users an escape hatch when Continue isn't necessarily properly configured for their model.

In general I don't see a strong argument in favor of allowing users to disable a given capability by omission in this array. If we really want that behavior I'd argue we should just add an additional config titled disallowed_capabilities - more verbose but less confusing.