Skip to content

[WIP]feat: add multiproc async #224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from
Draft

Conversation

kohlisid
Copy link
Contributor

@kohlisid kohlisid commented May 20, 2025

UDS Mode

(venv11) skohli@macos-JQWR9T560R async_multiproc_map % kubectl logs simple-pipeline-mult-0-ssrln -f | head -n 10 
+ python example.py
2025-05-21 23:03:15 INFO     Async GRPC Server listening on: unix:///var/run/numaflow/multiproc2.sock with max threads: 4
2025-05-21 23:03:15 INFO     Async GRPC Server listening on: unix:///var/run/numaflow/multiproc0.sock with max threads: 4
2025-05-21 23:03:15 INFO     Async GRPC Server listening on: unix:///var/run/numaflow/multiproc1.sock with max threads: 4

TCP Mode

(venv11) skohli@macos-JQWR9T560R async_multiproc_map % kubectl logs simple-pipeline-mult-0-wi104 -f | head -n 10 
+ python example.py
2025-05-21 23:04:44 INFO     Async GRPC Server listening on: 0.0.0.0:42285 with max threads: 4
2025-05-21 23:04:44 INFO     Async GRPC Server listening on: 0.0.0.0:35685 with max threads: 4
2025-05-21 23:04:44 INFO     Async GRPC Server listening on: 0.0.0.0:33085 with max threads: 4

kohlisid added 4 commits May 19, 2025 23:01
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
Copy link

codecov bot commented May 20, 2025

Codecov Report

Attention: Patch coverage is 52.94118% with 40 lines in your changes missing coverage. Please review.

Project coverage is 92.80%. Comparing base (6ccd49e) to head (62b7a85).

Files with missing lines Patch % Lines
pynumaflow/mapper/async_multiproc_server.py 34.69% 32 Missing ⚠️
pynumaflow/mapper/_servicer/_async_servicer.py 70.00% 5 Missing and 1 partial ⚠️
...numaflow/reducestreamer/servicer/async_servicer.py 33.33% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #224      +/-   ##
==========================================
- Coverage   94.19%   92.80%   -1.39%     
==========================================
  Files          58       59       +1     
  Lines        2359     2419      +60     
  Branches      119      124       +5     
==========================================
+ Hits         2222     2245      +23     
- Misses        100      136      +36     
- Partials       37       38       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

kohlisid added 7 commits May 20, 2025 14:12
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid changed the title feat: add multiproc async [WIP]feat: add multiproc async May 21, 2025
Signed-off-by: Sidhant Kohli <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant