Skip to content

VSCode llamafile Support Is Broken #5530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
3 tasks done
ipaddicting opened this issue May 6, 2025 · 4 comments · May be fixed by #5726
Open
3 tasks done

VSCode llamafile Support Is Broken #5530

ipaddicting opened this issue May 6, 2025 · 4 comments · May be fixed by #5726
Assignees
Labels
ide:vscode Relates specifically to VS Code extension kind:bug Indicates an unexpected problem or unintended behavior os:mac Happening specifically on Mac

Comments

@ipaddicting
Copy link

ipaddicting commented May 6, 2025

Before submitting your bug report

Relevant environment info

- OS: macOS
- Continue version: v1.0.8
- IDE version: VSCode 1.99.3
- Model: llamafile v0.9.2
- config:
  
  "models": [
    {
      "title": "Qwen 2.5 Coder 32B",
      "model": "qwen2.5-coder:32b",
      "provider": "llamafile",
      "apiBase": "http://127.0.0.1:8001"
    }
  ],

Description

The support of llamafile on VSCode is broken.

I rolled back the Continue to v1.0.1, and it just works fine, log output as following:

{"function":"log_server_request","level":"INFO","line":2842,"method":"POST","msg":"request","params":{},"path":"/completion","remote_addr":"127.0.0.1","remote_port":57485,"status":200,"tid":"5188741632","timestamp":1746515946}

To reproduce

  1. Open chat
  2. Type who are you
  3. Press Enter

Log output

Error message from continue:

Unexpected token 'F', "File Not Found" is not valid JSON

And the error message from llamafile:

{"function":"log_server_request","level":"INFO","line":2842,"method":"POST","msg":"request","params":{},"path":"/completions","remote_addr":"127.0.0.1","remote_port":58247,"status":404,"tid":"5188744736","timestamp":1746516335}

Looks like the path of request changed, from "path":"/completion" to path":"/completions".

@dosubot dosubot bot added ide:vscode Relates specifically to VS Code extension kind:bug Indicates an unexpected problem or unintended behavior labels May 6, 2025
@ipaddicting
Copy link
Author

ipaddicting commented May 6, 2025

The bug was introduced by this pull request Update LlamaCpp.ts #5030 on v1.0.6-vscode, because Llamafile is extends LlamaCpp.

@dosubot dosubot bot added the os:mac Happening specifically on Mac label May 7, 2025
@ipaddicting
Copy link
Author

ipaddicting commented May 19, 2025

Just check the updated documentation from llama.cpp, the URL remains completion: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#post-completion-given-a-prompt-it-returns-the-predicted-completion.

/v1/completions is for OAI-compatible clients only, but in LlamaCpp.ts there is no /v1 presented, and I don't think it should.

@RomneyDa
Copy link
Collaborator

@ipaddicting thanks for the quick turnaround on this

@RomneyDa
Copy link
Collaborator

@sestinj assigning to you since related pr #5726 reverts #5030

@RomneyDa RomneyDa assigned sestinj and unassigned RomneyDa May 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ide:vscode Relates specifically to VS Code extension kind:bug Indicates an unexpected problem or unintended behavior os:mac Happening specifically on Mac
Projects
Status: Todo
Development

Successfully merging a pull request may close this issue.

3 participants