[Doc] Add inplace weights loading example #19640

22quinn · 2025-06-14T06:05:36Z

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Provide an example of inplace weights loading feature added in #18745 #19884

Test Plan

python examples/offline_inference/inplace_weights_loading.py

Test Result

INFO 06-19 18:10:17 [gpu_model_runner.py:1695] Starting to load model facebook/opt-125m...
INFO 06-19 18:10:19 [gpu_model_runner.py:1705] Model was already initialized. Loading weights inplace...
INFO 06-19 18:10:19 [weight_utils.py:292] Using model weights format ['*.bin']
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  3.77it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  3.77it/s]

INFO 06-19 18:10:19 [default_loader.py:272] Loading weights took 0.27 seconds
INFO 06-19 18:10:20 [gpu_model_runner.py:1724] Model loading took 0.0000 GiB and 0.404207 seconds
Adding requests: 100%|██████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 237.13it/s]
Processed prompts: 100%|█████| 4/4 [00:00<00:00, 18.14it/s, est. speed input: 117.96 toks/s, output: 290.34 toks/s]

LLM Outputs:
------------------------------------------------------------
Prompt:    'Hello, my name is'
Output:    ' Joel, my dad is my friend and we are in a relationship. I am'
------------------------------------------------------------
Prompt:    'The president of the United States is'
Output:    ' reportedly being investigated by the FBI over the allegations that he broke campaign finance laws in'
------------------------------------------------------------
Prompt:    'The capital of France is'
Output:    ' all set to open a new gateway to the most notorious terror camps in Europe,'
------------------------------------------------------------
Prompt:    'The future of AI is'
Output:    ' in their hands. We cannot prevent the future.\nAgreed.  '
------------------------------------------------------------


INFO 06-19 18:10:37 [gpu_model_runner.py:1695] Starting to load model facebook/opt-125m...
INFO 06-19 18:10:37 [gpu_model_runner.py:1705] Model was already initialized. Loading weights inplace...
INFO 06-19 18:10:37 [weight_utils.py:292] Using model weights format ['*.bin']
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  3.43it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  3.42it/s]

INFO 06-19 18:10:38 [default_loader.py:272] Loading weights took 0.30 seconds
INFO 06-19 18:10:38 [gpu_model_runner.py:1724] Model loading took 0.0000 GiB and 0.391163 seconds
INFO 06-19 18:10:38 [async_llm.py:270] Added request 0.

LLM Outputs:
------------------------------------------------------------
Prompt:    'What is LLM?'
Output:    '\n\nLLM is a term used to describe the process of learning and developing'
------------------------------------------------------------

(Optional) Documentation Update

Signed-off-by: 22quinn <[email protected]>

github-actions · 2025-06-14T06:05:44Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Summary of Changes

Hello @22quinn, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new example script to the examples/offline_inference directory. The purpose is to provide a clear, runnable demonstration of how to use the recently added feature that allows initializing the vLLM engine with dummy weights and then loading the real model weights later, inplace, via an RPC call. This helps users understand and utilize this advanced loading capability.

Highlights

New Example Script: Adds a new Python script (examples/offline_inference/inplace_weights_loading.py) specifically designed to showcase the inplace weights loading feature.
Demonstrates Inplace Loading: The example initializes the LLM with load_format="dummy", then updates the engine's configuration to load_format="auto" and triggers the actual model loading via llm.collective_rpc("load_model").
Verification: Includes a step to run inference after loading the real weights to verify the process was successful.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configureGemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request adds an example demonstrating the inplace weights loading feature. The suggestions focus on removing unrelated code and enhancing comments for better clarity.

gemini-code-assist · 2025-06-14T06:06:56Z

examples/offline_inference/inplace_weights_loading.py

+# def aync_main():
+#     engine = AsyncLLMEngine.from_engine_args(engine_args)
+#     example_input = {
+#         "prompt": "What is LLM?",
+#         "stream": False, # assume the non-streaming case
+#         "temperature": 0.0,
+#         "request_id": 0,
+#     }
+#     # start the generation
+#     results_generator = engine.generate(
+#     example_input["prompt"],
+#     SamplingParams(temperature=example_input["temperature"]),
+#     example_input["request_id"])
+#     # get the results
+#     final_output = None
+#     async for request_output in results_generator:
+#         if await request.is_disconnected():
+#             # Abort the request if the client disconnects.
+#             await engine.abort(request_id)
+#             # Return or raise an error
+#             ...
+#         final_output = request_output


This commented-out code block appears unrelated to the example's purpose and could be removed to improve clarity.

gemini-code-assist · 2025-06-14T06:06:56Z

examples/offline_inference/inplace_weights_loading.py

+    llm.llm_engine.vllm_config.load_config.load_format = "auto"
+    llm.collective_rpc("load_model")


Consider adding more detail to this comment, explaining the two-step process: updating load_format and then calling collective_rpc.

Suggested change

llm.llm_engine.vllm_config.load_config.load_format = "auto"

llm.collective_rpc("load_model")

# Now load real weights inplace.

# 1. Update load_format to 'auto'.

# 2. Trigger model loading with collective_rpc.

examples/offline_inference/inplace_weights_loading.py

Signed-off-by: 22quinn <[email protected]>

yeqcharlotte · 2025-06-20T05:25:50Z

examples/offline_inference/skip_loading_weights_in_engine_init.py

+    llm = LLM(
+        model="facebook/opt-125m",
+        load_format="dummy",
+        enforce_eager=True,


to demonstrate it functions ok, i think we'd need tp2 at least

yeqcharlotte · 2025-06-20T05:37:27Z

examples/offline_inference/skip_loading_weights_in_engine_init.py

+sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
+
+
+async def aync_main():


i don't think we need to keep a sync and an async version. as the usage of those collective_rpcs are almost the same.

if you want to be comprehensive, it'll be nicer to follow one of the existing rlhf examples like https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/rlhf_colocate.py

inplace weights loading example

9b2547d

Signed-off-by: 22quinn <[email protected]>

gemini-code-assist bot reviewed Jun 14, 2025

View reviewed changes

mergify bot added the documentation Improvements or additions to documentation label Jun 14, 2025

gemini-code-assist bot reviewed Jun 14, 2025

View reviewed changes

22quinn changed the title ~~[Docs] Add inplace weights loading example~~ [Doc] Add inplace weights loading example Jun 14, 2025

houseroad reviewed Jun 14, 2025

View reviewed changes

examples/offline_inference/inplace_weights_loading.py Outdated Show resolved Hide resolved

22quinn added 5 commits June 14, 2025 23:00

Merge branch 'main' into inplace-example

bb960c1

Signed-off-by: 22quinn <[email protected]>

draft

0a884c6

Signed-off-by: 22quinn <[email protected]>

Merge branch 'main' into inplace-example

2fb9260

Signed-off-by: 22quinn <[email protected]>

update

9f1991c

Signed-off-by: 22quinn <[email protected]>

update

a85f92e

Signed-off-by: 22quinn <[email protected]>

22quinn mentioned this pull request Jun 20, 2025

[Core] Add update_load_config RPC method #19884

Open

4 tasks

22quinn marked this pull request as ready for review June 20, 2025 01:18

22quinn requested review from yeqcharlotte and youkaichao June 20, 2025 01:19

yeqcharlotte reviewed Jun 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Doc] Add inplace weights loading example #19640

[Doc] Add inplace weights loading example #19640

Uh oh!

22quinn commented Jun 14, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Jun 14, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jun 14, 2025

Uh oh!

gemini-code-assist bot Jun 14, 2025

Uh oh!

Uh oh!

yeqcharlotte Jun 20, 2025

Uh oh!

yeqcharlotte Jun 20, 2025

Uh oh!

Uh oh!

		llm.llm_engine.vllm_config.load_config.load_format = "auto"
		llm.collective_rpc("load_model")

-    llm.llm_engine.vllm_config.load_config.load_format = "auto"
-    llm.collective_rpc("load_model")
+    # Now load real weights inplace.
+    # 1. Update load_format to 'auto'.
+    # 2. Trigger model loading with collective_rpc.

		sampling_params = SamplingParams(temperature=0.8, top_p=0.95)


		async def aync_main():

Uh oh!

[Doc] Add inplace weights loading example #19640

Are you sure you want to change the base?

[Doc] Add inplace weights loading example #19640

Uh oh!

Conversation

22quinn commented Jun 14, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Essential Elements of an Effective PR Description Checklist

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

github-actions bot commented Jun 14, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jun 14, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yeqcharlotte Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

yeqcharlotte Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

22quinn commented Jun 14, 2025 •

edited by github-actions bot

Loading