[BUG]: Error loading the LLava model

[aropb]

Models:
https://huggingface.co/benxh/Qwen2.5-VL-7B-Instruct-GGUF
https://huggingface.co/KBlueLeaf/llama3-llava-next-8b-gguf (from here #897)
https://huggingface.co/second-state/Llava-v1.5-7B-GGUF

Error:
External component has thrown an exception.
System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
at LLama.Native.SafeLlavaModelHandle.clip_model_load(String mmProj, Int32 verbosity)
at LLama.Native.SafeLlavaModelHandle.LoadFromFile(String modelPath, Int32 verbosity)
at LLama.LLavaWeights.LoadFromFile(String mmProject)

What could be the problem?
I wanted to use a multimodal model to convert image2text.

Environment & Configuration

• Operating system: Windows
• .NET runtime version: NET 9.0.3
• LLamaSharp version: 0.23.0
• CUDA version (if you are using cuda backend): CPU
• CPU & GPU device: CPU

[martindevans]

Does the model you're using work with the corresponding version of llama.cpp? Maybe also test in the previous version of LLamaSharp, to check for regressions.

[aropb]

I'll try. Can you tell me if llava should work with any multimodal models?

I also noticed that the context is created only through llama weight, why is that?

[aropb]

Does the model you're using work with the corresponding version of llama.cpp? Maybe also test in the previous version of LLamaSharp, to check for regressions.

0.21.0 - the same mistake

[aropb]

Does the model you're using work with the corresponding version of llama.cpp?

CPU

Yes, loaded!

[SignalRT]

I tested the master with the model used in the unit test successfully.

Tested on Windows with CUDA.

[aropb]

and on the CPU?

[aropb]

I tested the master with the model used in the unit test successfully.

Models/llava-v1.6-mistral-7b.Q3_K_XS.gguf ?

[SignalRT]

As I explain I tested in GPU. The unit test are running successfully so it should work. I'm talking about Llava. Qwen-VL is not tested, and should not work.

[aropb]

Qwen-VL?

[aropb]

Does LLava not work on the CPU?

[SignalRT]

Qwen-VL?

Is that LlamaSharp documentation?

[aropb]

mmproj-model-f16.gguf - works!
llava-v1.6-mistral-7b-Q5_K_S.gguf - dont works!
https://huggingface.co/bartowski/Qwen2-VL-7B-Instruct-GGUF - dont works!

[aropb]

Is that LlamaSharp documentation?

Yes
https://github.com/ggml-org/llama.cpp

[aropb]

Should the LLava model be mmproj only?
Is it really impossible to use even Qwen2-VL for image2text?

Maybe I don't understand correctly how to work with VLM.

[SignalRT]

mmproj-model-f16.gguf - works! llava-v1.6-mistral-7b-Q5_K_S.gguf - dont works! https://huggingface.co/bartowski/Qwen2-VL-7B-Instruct-GGUF - dont works!

You need both files. Check https://scisharp.github.io/LLamaSharp/0.23.0/QuickStart/ and the Llava example in the examples project

[SignalRT]

Is that LlamaSharp documentation?

Yes https://github.com/ggml-org/llama.cpp

That's llama.cpp documentation

Is that LlamaSharp documentation?

Yes https://github.com/ggml-org/llama.cpp

That's llama.cpp documention. In the list of supported models in LlamaSharp documentation https://github.com/SciSharp/LLamaSharp qwen-vl is not in the list.

If I find some time I will test it.

[aropb]

Thanks. Yes, I get it.

[aropb]

If I find some time I will test it.

I found model (main + mmproj):
https://huggingface.co/second-state/Qwen2-VL-7B-Instruct-GGUF

and here many mmproj models:
https://huggingface.co/koboldcpp/mmproj/tree/main

All models are loaded, but the output is not working. The models from the example work, but they are weak and old. I would like to try Qwen2-VL or gemma3 (it doesn't work yet, apparently a new version is needed llama.cpp, ggml-org/llama.cpp#12344).

[SignalRT]

@aropb,

After conducting several tests and reviewing the current status of llama.cpp in relation to multimodal models, my understanding is as follows:

1. Qwen2-VL is supported but operates using its own CLI: qwen2vl-cli.cpp.
2. Gemma3 is still experimental and also requires its own CLI: gemma3-cli.cpp. Additional details can be found here.

Regarding LlamaSharp, only Llava and similar models are currently compatible. I don’t believe that replicating the work done in qwen2vl-cli or gemma3-cli would be the best approach. Instead, I recommend waiting for llama.cpp to introduce a vision API and then updating LlamaSharp’s multimodal support to integrate with that API.

[aropb]

Thanks.
Very useful information. I've been trying these models for the second day, really nothing works except llava :) And which of llava is considered the most recent and powerful?

Apparently it is, but it is weak by modern standards for Vision:
https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-gguf

[aropb]

Instead, I recommend waiting for llama.cpp to introduce a vision API and then updating LlamaSharp’s multimodal support to integrate with that API.

I haven't found any information about this, can you show me where they discuss it?

I found:
ggml-org/llama.cpp#9687
ggml-org/llama.cpp#11292

[SignalRT]

Instead, I recommend waiting for llama.cpp to introduce a vision API and then updating LlamaSharp’s multimodal support to integrate with that API.

I haven't found any information about this, can you show me where they discuss it?

I found: ggml-org/llama.cpp#9687 ggml-org/llama.cpp#11292

#1178 That's the information