[Misc][tool] Add a growth obj analyse tools to troubleshoot OOM issues #20024

MiaoChangyu · 2025-06-24T14:44:09Z

Purpose

Add an interface that uses the obj_graph component to detect object growth and troubleshoot OOM issues. By using this component, the following problems have been successfully solved.

#15043 (comment)

You can use it like torch_profile.

This tool can output the reference chain of the growing objects, as well as the types of the growing objects(dot).

You can use graphviz to conver dot to svg or npg

brew update
brew install graphviz

dot -Tsvg 20250610_124350_KVCacheBlock_refs.dot -o 20250610_124350_KVCacheBlock_refs.svg

dot -Tpng -Gdpi=300 20250610_124350_KVCacheBlock_refs.dot -o 20250610_124350_KVCacheBlock_refs.png

Such as:

In the future, I will support this feature in the V1 version.

Test Plan

In vLLM project, add an example.

Test Result

(Optional) Documentation Update

github-actions · 2025-06-24T14:44:17Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

MiaoChangyu · 2025-06-24T14:44:57Z

Teacher @WoosukKwon , please look here.

gemini-code-assist

Summary of Changes

Hello @MiaoChangyu, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new object graph analysis tool to help diagnose and resolve memory-related issues, particularly OOM errors. It adds necessary components and API endpoints to enable and control the analysis, providing valuable insights into object growth within the vLLM framework.

Highlights

Object Graph Analysis Tool: Introduces a new tool using the obj_graph component to detect object growth, aiding in troubleshooting OOM (Out of Memory) issues.
Environment Variable: Adds an environment variable VLLM_OBJ_GRAPH_DIR to enable object graph analysis and specify the output directory.
API Endpoints: Adds new API endpoints /start_object_graph and /stop_object_graph to control object graph analysis via API calls.
GrowingMemoryObjGraph Class: Introduces a GrowingMemoryObjGraph class in vllm/utils.py to handle the start and stop logic for object graph analysis, including generating .dot files and log files.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The pull request introduces object growth analysis tools to troubleshoot OOM issues by adding a new GrowingMemoryObjGraph class and integrating it into the vLLM framework. It also adds an example to demonstrate the usage of the new feature. The code generally looks good, but there are a few minor suggestions for improvement.

gemini-code-assist · 2025-06-24T14:46:49Z

vllm/utils.py

+        analysis_dir = os.path.join(self._obj_graph_dir, f"analysis_{current_date}")
+        os.makedirs(analysis_dir, exist_ok=True)


Consider adding a try-except block to handle potential OSError exceptions when creating the analysis directory, as directory creation can sometimes fail due to permissions or other issues.

Suggested change

analysis_dir = os.path.join(self._obj_graph_dir, f"analysis_{current_date}")

os.makedirs(analysis_dir, exist_ok=True)

try:

os.makedirs(analysis_dir, exist_ok=True)

except OSError as e:

logger.error(f"Failed to create directory {analysis_dir}: {e}")

return f"Failed to create directory: {e}"

gemini-code-assist · 2025-06-24T14:46:49Z

vllm/utils.py

+
+        output_file_path = os.path.join(analysis_dir, "growing_memory_stats.log")
+        with open(output_file_path, 'w', encoding='utf-8') as f:
+            for line in output_lines:
+                f.write(line + '\n')


Consider adding a try-except block to handle potential OSError exceptions when writing to the output file, as file writing can sometimes fail due to permissions or other issues.

Suggested change

output_file_path = os.path.join(analysis_dir, "growing_memory_stats.log")

with open(output_file_path, 'w', encoding='utf-8') as f:

for line in output_lines:

f.write(line + '\n')

try:

with open(output_file_path, 'w', encoding='utf-8') as f:

for line in output_lines:

f.write(line + '\n')

except OSError as e:

logger.error(f"Failed to write to file {output_file_path}: {e}")

return f"Failed to write to file: {e}"

MiaoChangyu · 2025-06-24T15:15:23Z

Teacher @njhill , please look here. Thanks.

Signed-off-by: miaochangyu <[email protected]>

MiaoChangyu · 2025-06-24T15:36:58Z

Teacher @zhuohan123 , please look here, thanks you~

Signed-off-by: miaochangyu <[email protected]>

mergify · 2025-06-25T02:42:03Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @MiaoChangyu.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

…j_analyse_tools # Conflicts: # vllm/utils.py

Signed-off-by: miaochangyu <[email protected]>

mergify · 2025-06-25T03:37:56Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @MiaoChangyu.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

MiaoChangyu requested review from aarnphm, zhuohan123, youkaichao, alexm-redhat, comaniac and njhill as code owners June 24, 2025 14:44

mergify bot added documentation Improvements or additions to documentation ci/build frontend labels Jun 24, 2025

gemini-code-assist bot reviewed Jun 24, 2025

View reviewed changes

miaochangyu added 3 commits June 24, 2025 23:31

add growing object graph util to analyse memory leak and OOM

f7d402f

Signed-off-by: miaochangyu <[email protected]>

add offline example

4c47615

Signed-off-by: miaochangyu <[email protected]>

fix Code Review comment

ff0be92

Signed-off-by: miaochangyu <[email protected]>

MiaoChangyu force-pushed the add_growth_obj_analyse_tools branch from 53fdae2 to ff0be92 Compare June 24, 2025 15:31

fix pre-commit pipeline

d0b65b9

Signed-off-by: miaochangyu <[email protected]>

mergify bot added the needs-rebase label Jun 25, 2025

Merge branch 'main' of github.com:MiaoChangyu/vllm into add_growth_ob…

16aa44d

…j_analyse_tools # Conflicts: # vllm/utils.py

mergify bot removed the needs-rebase label Jun 25, 2025

miaochangyu added 2 commits June 25, 2025 11:08

fix pre-commit pipeline

ff621b3

Signed-off-by: miaochangyu <[email protected]>

fix pre-commit pipeline

dd497d8

Signed-off-by: miaochangyu <[email protected]>

mergify bot added the needs-rebase label Jun 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Misc][tool] Add a growth obj analyse tools to troubleshoot OOM issues #20024

[Misc][tool] Add a growth obj analyse tools to troubleshoot OOM issues #20024

MiaoChangyu commented Jun 24, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Jun 24, 2025

Uh oh!

MiaoChangyu commented Jun 24, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jun 24, 2025

Uh oh!

gemini-code-assist bot Jun 24, 2025

Uh oh!

MiaoChangyu commented Jun 24, 2025

Uh oh!

MiaoChangyu commented Jun 24, 2025

Uh oh!

mergify bot commented Jun 25, 2025

Uh oh!

mergify bot commented Jun 25, 2025

Uh oh!

Uh oh!

		analysis_dir = os.path.join(self._obj_graph_dir, f"analysis_{current_date}")
		os.makedirs(analysis_dir, exist_ok=True)

Uh oh!

[Misc][tool] Add a growth obj analyse tools to troubleshoot OOM issues #20024

Are you sure you want to change the base?

[Misc][tool] Add a growth obj analyse tools to troubleshoot OOM issues #20024

Conversation

MiaoChangyu commented Jun 24, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

github-actions bot commented Jun 24, 2025

Uh oh!

MiaoChangyu commented Jun 24, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

MiaoChangyu commented Jun 24, 2025

Uh oh!

MiaoChangyu commented Jun 24, 2025

Uh oh!

mergify bot commented Jun 25, 2025

Uh oh!

mergify bot commented Jun 25, 2025

Uh oh!

Uh oh!

MiaoChangyu commented Jun 24, 2025 •

edited by github-actions bot

Loading