Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enhance: make pchannel level flusher #39275

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

enhance: make pchannel level flusher #39275

wants to merge 2 commits into from

Conversation

chyezh
Copy link
Contributor

@chyezh chyezh commented Jan 15, 2025 •
edited
Loading

issue: #38399

  • Add a pchannel level checkpoint for flush processing
  • Refactor the recovery of flushers of wal
  • make a shared wal scanner first, then make multi datasyncservice on it

@sre-ci-robot sre-ci-robot added area/internal-api area/test sig/testing test/integration integration test size/XXL Denotes a PR that changes 1000+ lines. labels Jan 15, 2025
@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh
To complete the pull request process, please assign congqixia after the PR has been reviewed.
You can assign the PR to them by writing /assign @congqixia in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify mergify bot added the dco-passed DCO check passed. label Jan 15, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 15, 2025

@chyezh

Invalid PR Title Format Detected

Your PR submission does not adhere to our required standards. To ensure clarity and consistency, please meet the following criteria:

  1. Title Format: The PR title must begin with one of these prefixes:
  • feat: for introducing a new feature.
  • fix: for bug fixes.
  • enhance: for improvements to existing functionality.
  • test: for add tests to existing functionality.
  • doc: for modifying documentation.
  • auto: for the pull request from bot.
  1. Description Requirement: The PR must include a non-empty description, detailing the changes and their impact.

Required Title Structure:

[Type]: [Description of the PR]

Where Type is one of feat, fix, enhance, test or doc.

Example:

enhance: improve search performance significantly

Please review and update your PR to comply with these guidelines.

@codecov Codecov
Copy link

codecov bot commented Jan 15, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.39%. Comparing base (05ac404) to head (35c53a9).
Report is 3 commits behind head on master.

❌ Your project check has failed because the head coverage (69.39%) is below the target coverage (77.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #39275      +/-   ##
==========================================
- Coverage   69.40%   69.39%   -0.02%     
==========================================
  Files         302      302              
  Lines       27065    27070       +5     
==========================================
  Hits        18785    18785              
- Misses       8280     8285       +5
Components Coverage Δ
Client ∅ <ø> (∅)
Core 69.39% <ø> (-0.02%) ⬇️
Go ∅ <ø> (∅)

see 2 files with indirect coverage changes

@chyezh chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from 17e4298 to 44e5e19 Compare January 23, 2025 09:02
@chyezh chyezh mentioned this pull request Jan 22, 2025
Open
10 tasks
@chyezh chyezh added this to the 2.6.0 milestone Jan 23, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 23, 2025

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

@chyezh chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from 44e5e19 to 22ad3ac Compare January 23, 2025 10:03
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 23, 2025

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 23, 2025

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@chyezh chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from 22ad3ac to e2f0c28 Compare February 5, 2025 08:14
@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 5, 2025

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 5, 2025

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@chyezh
Copy link
Contributor Author

chyezh commented Feb 5, 2025

/run-cpu-e2e

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 5, 2025

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

@chyezh
enhance: erase the rpc level when wal is located at same node
d208c22 
- Make the wal scanner interface same with streaming scanner.
- Use wal if the wal is located at current node.
- Otherwise fallback the old logic.

Signed-off-by: chyezh <[email protected]>
@chyezh chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from e2f0c28 to 1c8d96c Compare February 5, 2025 11:04
@chyezh
enhance: make pchannel level flusher
35c53a9 
- Add a pchannel level checkpoint for flush processing
- Refactor the recovery of flushers of wal
- make a shared wal scanner first, then make multi datasyncservice on it

Signed-off-by: chyezh <[email protected]>
@chyezh chyezh force-pushed the enhance_make_pchannel_lv_flusher branch from 1c8d96c to 35c53a9 Compare February 5, 2025 12:35
@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 5, 2025

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 5, 2025

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

@congqixia congqixia Awaiting requested review from congqixia

Assignees
No one assigned
Labels
area/internal-api area/test dco-passed DCO check passed. do-not-merge/invalid-pr-format sig/testing size/XXL Denotes a PR that changes 1000+ lines. test/integration integration test
Projects
None yet
Milestone
2.6.0
Development

Successfully merging this pull request may close these issues.
2 participants
@chyezh @sre-ci-robot