Add Llama4 Multi-Modal Support #382

vbaddi · 2025-04-29T10:41:32Z

No description provided.

vbaddi · 2025-04-30T09:48:42Z

QEfficient/transformers/models/llama4/modeling_llama4.py

+    def __qeff_init__(self):
+        self.rotary_embedding = QEffLlama4VisionRotaryEmbedding(config=self.config)
+
+    def forward(


do we need this forward? remove

vbaddi · 2025-04-30T09:49:10Z

QEfficient/transformers/models/llama4/modeling_llama4.py

-            17,  # constants.INTERN_NUM_PATCHES,
-            14,  # constants.INTERN_FEATURE_SIZE,
+        inputs_shapes["vision_embeds"] = (
+            1,  # constants.INTERN_NUM_PATCHES,


move to constants? or compute based on the code.

quic-sanising · 2025-06-04T23:09:15Z

pyproject.toml

@@ -19,8 +19,8 @@ classifiers = [
 ]
 requires-python = ">=3.8,<3.11"
 dependencies = [
-    "transformers==4.50.0",
-    "huggingface-hub==0.27.0",
+    "transformers==4.51.0",


We have moved transformers to 4.51.3 in vLLM

QEfficient/base/pytorch_transforms.py

quic-rishinr · 2025-06-06T08:13:52Z

QEfficient/base/modeling_qeff.py

@@ -46,6 +46,7 @@ class QEFFBaseModel(ABC):
    def _transform_names(cls) -> List[str]:
        return [x.__name__ for x in cls._pytorch_transforms + cls._onnx_transforms]

+    @append_tranform


Instead of adding a decorator to the constructor define a method for applying pytorch transforms, pass a copy of cls._pytorch_transforms. You can add the decorator for that method.

quic-rishinr · 2025-06-06T08:17:36Z

QEfficient/transformers/models/llama4/modeling_llama4.py

+        real, imag = torch.cos(freqs), torch.sin(freqs)
+        freqs_cis = torch.stack([real, imag], dim=-1)
+
+        #    freqs_cis = torch.view_as_complex(freqs_cis.contiguous())


remove commented lines

quic-rishinr · 2025-06-06T08:18:19Z

QEfficient/transformers/models/llama4/modeling_llama4.py

+        final = shared_out + expert_out  # restore [B,S,H]
+        return final.view(B, S, H), router_logits
+
+    # ------------------- Gather based, weights as activation approach ---------------


Do we need this?

Ig no, will remove the other commented approaches.

quic-rishinr · 2025-06-06T08:22:52Z

QEfficient/transformers/models/pytorch_transforms.py

@@ -1,13 +1,14 @@
 # -----------------------------------------------------------------------------
 #
-# Copyright (c) 2025 Qualcomm Innovation Center, Inc. All rights reserved.
+# Copyright (c) 2024 Qualcomm Innovation Center, Inc. All rights reserved.


please update the year

quic-rishinr · 2025-06-06T08:23:26Z

QEfficient/transformers/models/pytorch_transforms.py

@@ -49,29 +50,28 @@
    GraniteModel,
    GraniteRMSNorm,
 )


Why GraniteMoe is removed?

quic-rishinr · 2025-06-06T08:23:39Z

QEfficient/transformers/models/pytorch_transforms.py

-    QEffGraniteMoeParallelExperts,
-    QEffGraniteMoeRotaryEmbedding,
-    QEffGraniteMoeTopKGating,
+from QEfficient.transformers.models.internvl.modeling_internvl import (


Same as above

quic-rishinr · 2025-06-06T08:24:40Z

QEfficient/transformers/models/pytorch_transforms.py

@@ -401,6 +404,8 @@ class KVCacheTransform(ModuleMappingTransform):
    @classmethod
    def apply(cls, model: nn.Module) -> Tuple[nn.Module, bool]:
        model, transformed = super().apply(model)
+        # FIXME: see if we can merge into _module_mapping dict
+        transformers.cache_utils.DynamicCache.update = QEffDynamicCache.update


Do we need it here? since we will be going with HybridChunkCache?

quic-rishinr · 2025-06-09T06:11:50Z

@mohiso22 please rebase the PR. add test and add the model to validated model list

examples/llama4_lm_example.py

vbaddi · 2025-06-10T06:14:36Z

QEfficient/transformers/models/modeling_auto.py


        lang_inputs = {k: v for k, v in inputs.items() if k not in vision_inputs}
        lang_inputs["position_ids"] = np.where(
            lang_inputs.pop("attention_mask"), np.arange(padded_len), -1
        )  # Need to use -1 as position_ids for invalid tokens

+        NOT_MLLAMA = hasattr(self.model.config, "model_type") and self.model.config.model_type != "mllama"
+        if NOT_MLLAMA:
+            lang_inputs["index"] = np.array([[0]])


@mohiso22 can we rename the index to something which is more readable? image_idx?

quic-rishinr · 2025-06-10T06:34:44Z

QEfficient/transformers/models/llama4/modeling_llama4.py

+        self.language_model = self.model.language_model
+        self.config = self.model.config
+
+    def forward(self, input_ids, vision_embeds, position_ids, index, past_key_values):


Can you add a detailed the doc string with the interleaving logic we are using here.

Please renaming the index variable to something more descriptive, such as vision_start_index the current name index is too generic.

quic-rishinr · 2025-06-10T08:08:24Z

QEfficient/transformers/models/llama4/modeling_llama4.py

+    def forward(self, input_ids, vision_embeds, position_ids, index, past_key_values):
+        inputs_embeds = self.model.language_model.get_input_embeddings()(input_ids)
+        selected = input_ids == self.model.config.image_token_index
+        indices1 = selected.to(torch.int64).cumsum(1) - 1


instead of indices0 and indices1 please use more descriptive variable names

quic-rishinr · 2025-06-10T08:13:40Z

QEfficient/transformers/models/llama4/modeling_llama4.py

+    def get_qeff_language_decoder(self):
+        return QEffLlama4DecoderWrapper(self)
+
+    def forward(self, input_ids, position_ids, pixel_values, index, past_key_values):


please use more descriptive variable name for index

quic-rishinr · 2025-06-10T08:13:54Z

QEfficient/transformers/models/llama4/modeling_llama4.py

+        vision_flat = image_features.view(-1, image_features.size(-1))
+        projected_vision_flat = self.multi_modal_projector(vision_flat)
+        selected = input_ids == self.config.image_token_index
+        indices1 = selected.to(torch.int64).cumsum(1) - 1


Same as above

quic-rishinr · 2025-06-10T08:17:19Z

QEfficient/transformers/models/modeling_auto.py


        lang_inputs = {k: v for k, v in inputs.items() if k not in vision_inputs}
        lang_inputs["position_ids"] = np.where(
            lang_inputs.pop("attention_mask"), np.arange(padded_len), -1
        )  # Need to use -1 as position_ids for invalid tokens

+        NOT_MLLAMA = hasattr(self.model.config, "model_type") and self.model.config.model_type != "mllama"


its not a constant why we are using uppercase here?

ok will change to smallercase

quic-rishinr · 2025-06-10T08:23:11Z

examples/llama4_lm_example.py

@@ -0,0 +1,50 @@
+# -----------------------------------------------------------------------------


Remove this file. Lets maintain only one example different inputs, Text only, text + image, text + multiple images

Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

…ample Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

…mple files of llama4 as its now same as other Signed-off-by: Amit Raj <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Signed-off-by: Amit Raj <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Signed-off-by: Onkar Chougule <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

…_rope Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Signed-off-by: Amit Raj <[email protected]>

Signed-off-by: Rishin Raj <[email protected]> Signed-off-by: Amit Raj <[email protected]>

vbaddi added the model-enablement label Apr 29, 2025

vbaddi requested a review from mohiso22 April 29, 2025 10:41

vbaddi assigned mohiso22 Apr 29, 2025

vbaddi requested review from quic-rishinr and ochougul as code owners April 29, 2025 10:41

quic-rishinr marked this pull request as draft April 30, 2025 05:37

vbaddi commented Apr 30, 2025

View reviewed changes

quic-amitraj force-pushed the add_llama4 branch 2 times, most recently from 75242b2 to 1b3dd96 Compare May 8, 2025 10:05

quic-sanising reviewed Jun 4, 2025

View reviewed changes

quic-hemagnih added the 1.20.0 label Jun 5, 2025

quic-rishinr requested changes Jun 6, 2025

View reviewed changes

mohiso22 force-pushed the add_llama4 branch 2 times, most recently from 5d3870d to 77afbbc Compare June 9, 2025 10:22

quic-rishinr requested changes Jun 9, 2025

View reviewed changes

examples/llama4_lm_example.py Outdated Show resolved Hide resolved

vbaddi commented Jun 10, 2025

View reviewed changes

vbaddi marked this pull request as ready for review June 10, 2025 06:16

vbaddi requested review from quic-hemagnih and quic-amitraj as code owners June 10, 2025 06:16

quic-rishinr requested changes Jun 10, 2025

View reviewed changes

quic-amitraj force-pushed the add_llama4 branch 2 times, most recently from ee23c3a to 1066b4f Compare June 10, 2025 15:09

vbaddi and others added 7 commits June 10, 2025 15:10

Add Llama4 Multi-Modal Support

ceee86a

Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

nit: modeling changes

693e474

Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Adding Vision Part and Chunking (#383)

ed85612

Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

nit: update moe implementation and add sample export/compile script

77df9d2

Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

nit: fix linter for example script

b5eb805

Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

nit: update pytorch transforms to map Llama4TextExperts

eface81

Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

nit: update modeling with new freq apply computation and sample mm ex…

b1eae7e

…ample Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

vbaddi and others added 19 commits June 10, 2025 15:10

nit: update llama4 mm example script

5aca1ef

Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

nit: update modeling to avoid >2GiB issue in Onnx, rope max-position

91eca48

Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Added pytorch transform for the split_gate_up_weights and removed exa…

a661727

…mple files of llama4 as its now same as other Signed-off-by: Amit Raj <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Ruff Check and format

3cc5511

Signed-off-by: Amit Raj <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Minor fixes-1

6ca555e

Signed-off-by: Amit Raj <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Added logger for new transform

7482533

Signed-off-by: Amit Raj <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

fixed Llama4 MOE accuracy bug

a83f23f

Signed-off-by: Onkar Chougule <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Updating index method in Wrappers (#410)

f98f4af

Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

nit: add position_ids to attn_scales instead of cache_position in use…

1b36e90

…_rope Signed-off-by: vbaddi <[email protected]> Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Minor Fixes (#421)

1ebd9a8

Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Updating Specialization and modeling auto files

63b0b10

Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Fix for Multi Image Chunking

8e0a613

Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Adding SingleQPC

c91dd9f

Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Rebase and Minor Fixes

aefa2b0

Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Addressed Comments

ba30051

Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Addressed Comments

64ec975

Signed-off-by: Mohit Soni <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Addressed comments

d15fe69

Signed-off-by: Amit Raj <[email protected]>

Header and doc update

b7e1a46

Signed-off-by: Rishin Raj <[email protected]> Signed-off-by: Amit Raj <[email protected]>

Updated get_available_device_id method

d8a947a

Signed-off-by: Rishin Raj <[email protected]> Signed-off-by: Amit Raj <[email protected]>

quic-amitraj force-pushed the add_llama4 branch from 1066b4f to d8a947a Compare June 10, 2025 15:11

		@@ -0,0 +1,50 @@
		# -----------------------------------------------------------------------------

Add Llama4 Multi-Modal Support #382

Are you sure you want to change the base?

Add Llama4 Multi-Modal Support #382

Conversation

vbaddi commented Apr 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quic-rishinr commented Jun 9, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!