Issue with Custom LLM Configuration: 404 Error and Request Flow

[sowjanyagunupuru]

I have configured our custom-built LLM URL in the customOpenAImodels section of the settings.json file and selected it as the language model for the @coder agent. However, when I enter input and click "Send," I receive a 404 status code (no body)). Since no API keys are required to hit my URL, I have set apiKey to true.

"ai-features.openAiCustom.customOpenAiModels":[
{
"model": "pilot-demo",
"url": "https://pilot-demo.sample.com:2543/sample",
"id": "pilot-demo",
"apiKey": true,
"enableStreaming": true
}
]

When I use the same URL in Postman, it returns the expected response.

I would appreciate guidance on where the chat input is managed and how it can be modified before being included in the payload request to hit the API of the local LLM. Additionally, I would like to understand how the response is displayed.

Your help and suggestions would be invaluable in helping us move forward.

[sdirix]

Hi @sowjanyagunupuru,

This is where we hand over the key to the OpenAI SDK. We always need to set a key, as the SDK throws an error otherwise. By setting apiKey: true it will hand over the OpenAI key of your preferences / environment variable. If you don't need a key, then just omit the apiKey property.

Can you check whether your URL is offering an Open AI compatible endpoint, usually they end in v1, e.g. http://localhost:8000/v1. Can you post what request you did send via postman and what response you did receive?

[sowjanyagunupuru]

Okay, I removed the API key, but I am still getting the same 404 status code (with no body).

It seems that my LLM model URL is not offering an OpenAI-compatible endpoint.

Here is the POST request I sent via Postman, with the raw JSON placed in the body:

{
"a": "x",
"b": "y",
"payload": {
"input": "hi", // Here, the user input in the ai chat box is appended
"files": null
}
}
And the successful response I received in return is:

{
"result": "Hello! How can I assist you today?",
"Status": {
"code": "0000",
"text": "success"
}
}
Thank you for suggesting and supporting me throughout this!

[sdirix]

I see. The OpenAI client expects that there is a /chat/completions endpoint which returns a different response. See this documentation: https://platform.openai.com/docs/api-reference/chat/create. More complicated when you also want streaming.

I guess this endpoint is not exposed on your side, which is why you get the 404 errors.

You either need to make sure to expose an OpenAI compatible end point, for example via vllm, or you need to implement your own LanguageModel which knows how to interact with your endpoints. The LanguageModel can then be registered via a Theia extension and therefore made available in your Theia based application.