Shape Error in Llama4VisionMLP2

[priyammaz]

System Info

transformers/src/transformers/models/llama4/modeling_llama4.py

Line 1096 in d1b9236

self.fc2 = nn.Linear(config.projector_output_dim, config.projector_output_dim, bias=False)

Im pretty sure this should be:

self.fc2 = nn.Linear(config.projector_input_dim, config.projector_output_dim, bias=False)

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Nothing to reproduce, I just noticed a shape mistake, but i havent had a chance to run it personally yet!

Expected behavior

The output of fc1 should be the input of fc2, unless there is some hidden logic that I am missing

[priyammaz]

I now see the input and output are the same in the config, but still this can be a bug if someone changed that!

[Rocketknight1]

cc @ArthurZucker but this looks like a bug, yes!