Skip to content

[ISSUE #9254] Add CombineConsumeQueue to support CQ migration #9256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jun 3, 2025
Merged

[ISSUE #9254] Add CombineConsumeQueue to support CQ migration #9256

merged 3 commits into from
Jun 3, 2025

Conversation

qianye1001
Copy link
Contributor

Which Issue(s) This PR Fixes

Fixes #9254

Brief Description

How Did You Test This Change?

@qianye1001 qianye1001 marked this pull request as draft March 17, 2025 12:56
@codecov-commenter
Copy link

codecov-commenter commented Mar 18, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 51.68869% with 329 lines in your changes missing coverage. Please review.

Project coverage is 48.05%. Comparing base (c9ea749) to head (5620d38).
Report is 3 commits behind head on develop.

Files with missing lines Patch % Lines
...rocketmq/store/queue/CombineConsumeQueueStore.java 41.15% 140 Missing and 13 partials ⚠️
...apache/rocketmq/store/queue/ConsumeQueueStore.java 50.00% 77 Missing and 6 partials ⚠️
...rocketmq/store/queue/RocksDBConsumeQueueStore.java 47.05% 26 Missing and 1 partial ⚠️
...etmq/store/metrics/RocksDBStoreMetricsManager.java 0.00% 16 Missing ⚠️
...pache/rocketmq/store/dledger/DLedgerCommitLog.java 41.17% 6 Missing and 4 partials ⚠️
...main/java/org/apache/rocketmq/store/CommitLog.java 59.09% 5 Missing and 4 partials ⚠️
...ache/rocketmq/store/queue/RocksDBConsumeQueue.java 74.19% 8 Missing ⚠️
...ocketmq/broker/processor/AdminBrokerProcessor.java 0.00% 6 Missing ⚠️
...org/apache/rocketmq/store/DefaultMessageStore.java 89.58% 1 Missing and 4 partials ⚠️
...ache/rocketmq/store/config/MessageStoreConfig.java 75.00% 5 Missing ⚠️
... and 4 more
Additional details and impacted files 
@@              Coverage Diff              @@
##             develop    #9256      +/-   ##
=============================================
+ Coverage      47.96%   48.05%   +0.08%     
- Complexity     11908    11982      +74     
=============================================
  Files           1307     1308       +1     
  Lines          92021    92166     +145     
  Branches       11775    11789      +14     
=============================================
+ Hits           44139    44288     +149     
+ Misses         42401    42375      -26     
- Partials        5481     5503      +22

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@qianye1001 qianye1001 changed the title [ISSUE #9254] Refactor DefaultMessageStorage type/enhancement [ISSUE #9254] Refactor DefaultMessageStorage Mar 18, 2025
@qianye1001 qianye1001 marked this pull request as ready for review March 18, 2025 03:31
@qianye1001 qianye1001 changed the title [ISSUE #9254] Refactor DefaultMessageStorage [ISSUE #9254] Refactor CQ-related thread in DefaultMessageStorage Mar 18, 2025
@qianye1001 qianye1001 changed the title [ISSUE #9254] Refactor CQ-related thread in DefaultMessageStorage [ISSUE #9254] Refactor CQ-related in DefaultMessageStorage Mar 19, 2025
@qianye1001 qianye1001 force-pushed the Refactor-DefaultMessageStore branch 2 times, most recently from ad190f2 to 61ee2f4 Compare March 25, 2025 07:03
@qianye1001 qianye1001 force-pushed the Refactor-DefaultMessageStore branch 2 times, most recently from dc014b5 to a8996fa Compare April 10, 2025 07:03
lizhimins
lizhimins reviewed Apr 16, 2025
View reviewed changes
@qianye1001 qianye1001 force-pushed the Refactor-DefaultMessageStore branch from da78405 to 59ee17a Compare April 18, 2025 03:33
lollipopjin
lollipopjin reviewed Apr 22, 2025
View reviewed changes
lollipopjin
lollipopjin reviewed Apr 22, 2025
View reviewed changes
store/src/main/java/org/apache/rocketmq/store/queue/RocksDBConsumeQueue.java Outdated
@@ -480,4 +486,21 @@ public CqUnit nextAndRelease() {
}
}
}

public void initializeWithOffset(long offset) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is used for mount --> main in container mode? If so, add some java docs here.

store/src/main/java/org/apache/rocketmq/store/queue/RocksDBConsumeQueue.java Outdated

// update the max and min offset
if (offset > 0) {
this.consumeQueueStore.updateCqOffset(topic, queueId, 0L, offset - 1, true);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about use two methods for updating CQ offset for min and max offsets.

@qianye1001 qianye1001 force-pushed the Refactor-DefaultMessageStore branch from f56cea9 to 5165c2f Compare May 28, 2025 06:05
lizhimins
lizhimins reviewed May 29, 2025
View reviewed changes
store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java Outdated
final String storePath,
final int mappedFileSize,
final MessageStore messageStore,
final ConsumeQueueStore consumeQueueStore) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This format looks a bit strange

lizhimins
lizhimins reviewed May 29, 2025
View reviewed changes
store/src/main/java/org/apache/rocketmq/store/queue/CombineConsumeQueueStore.java Outdated

currentReadStore = getInnerStoreByString(messageStoreConfig.getCombineCQPreferCQType());
if (currentReadStore == null) {
log.error("CombineConsumeQueue choosePreferCQ fail, prefer={}", messageStoreConfig.getCombineCQPreferCQType());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

日志的风格，前后不太一致

@qianye1001 qianye1001 changed the title [ISSUE #9254] Refactor CQ-related in DefaultMessageStorage [ISSUE #9254] Add CombineConsumeQueue to support CQ migration Jun 3, 2025
lollipopjin
lollipopjin approved these changes Jun 3, 2025
View reviewed changes
Copy link
Contributor

@lollipopjin lollipopjin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now.

@lollipopjin lollipopjin merged commit 9d9431d into apache:develop Jun 3, 2025
10 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lizhimins lizhimins lizhimins left review comments

@lollipopjin lollipopjin lollipopjin approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Enhancement] Refactor CQ-related thread in DefaultMessageStorage
4 participants
@qianye1001 @codecov-commenter @lizhimins @lollipopjin