Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: multithread core workers #9591

Merged
merged 1 commit into from
Nov 12, 2024
Merged

core: multithread core workers #9591

merged 1 commit into from
Nov 12, 2024

Conversation

eckter
Copy link
Contributor

@eckter eckter commented Nov 6, 2024

Fix #9572

Configurable with a WORKER_THREADS=42 env variable, with a default set to the number of processors.

@eckter eckter requested a review from a team as a code owner November 6, 2024 09:21
@eckter eckter requested a review from Erashin November 6, 2024 09:21
@codecov-commenter
Copy link

codecov-commenter commented Nov 6, 2024

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 0% with 27 lines in your changes missing coverage. Please review.

Project coverage is 42.49%. Comparing base (f05eecf) to head (f228935).
Report is 2 commits behind head on dev.

Files with missing lines Patch % Lines
...re/src/main/java/fr/sncf/osrd/cli/WorkerCommand.kt 0.00% 27 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff              @@
##                dev    #9591      +/-   ##
============================================
- Coverage     42.50%   42.49%   -0.02%     
  Complexity     2272     2272              
============================================
  Files          1312     1312              
  Lines        105588   105608      +20     
  Branches       3304     3305       +1     
============================================
- Hits          44881    44873       -8     
- Misses        58754    58782      +28     
  Partials       1953     1953              
Flag Coverage Δ
core 74.95% <0.00%> (-0.11%) ⬇️
editoast 73.70% <ø> (-0.03%) ⬇️
front 18.79% <ø> (ø)
gateway 2.18% <ø> (ø)
osrdyne 3.28% <ø> (ø)
railjson_generator 87.49% <ø> (ø)
tests 86.74% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Khoyo
Copy link
Contributor

Khoyo commented Nov 6, 2024

Does this do parallel consume? Or does this launch a single task on the threadpool (and we would need to actually call consume multiple times on separate channels?)

@eckter eckter force-pushed the ech/parallelize-core branch 4 times, most recently from 8050c30 to 70e728b Compare November 6, 2024 10:14
@eckter
Copy link
Contributor Author

eckter commented Nov 6, 2024

Does this do parallel consume? Or does this launch a single task on the threadpool (and we would need to actually call consume multiple times on separate channels?)

The "consume" is only done on a single thread, but the processing itself is dispatched to the executor.

The alternative would be to create several channels and call several consumes, but I struggled to make it work and to have it actually work in parallel. (But I can try again later if the current solution bothers anyone)

Signed-off-by: Eloi Charpentier <[email protected]>
@eckter eckter force-pushed the ech/parallelize-core branch from 70e728b to f228935 Compare November 12, 2024 08:53
@eckter eckter enabled auto-merge November 12, 2024 08:54
@eckter eckter added this pull request to the merge queue Nov 12, 2024
Merged via the queue into dev with commit a0587f2 Nov 12, 2024
25 checks passed
@eckter eckter deleted the ech/parallelize-core branch November 12, 2024 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

core: when running as worker, we only use a single thread (per worker)
4 participants