-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Enhancement] adjust async-profile queue strategy #52236
base: main
Are you sure you want to change the base?
Conversation
Signed-off-by: Murphy <mofei@starrocks.com>
f3a8c28
to
1452385
Compare
Quality Gate failedFailed conditions See analysis details on SonarCloud Catch issues before they fail your Quality Gate with our IDE extension SonarLint |
Signed-off-by: Murphy <mofei@starrocks.com>
Signed-off-by: Murphy <mofei@starrocks.com>
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 6 / 6 (100.00%) file detail
|
[BE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
*/ | ||
private static final ThreadPoolExecutor EXECUTOR = | ||
ThreadPoolManager.newDaemonFixedThreadPool(Config.profile_process_threads_num, | ||
Integer.MAX_VALUE, "profile-worker", false); | ||
ThreadPoolManager.newDaemonCacheThreadPool( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so thread pool core size is 3/4 * all cpus, maximumPoolSize is 3 * All cpus? can you explain why you decide using these numbers
@@ -256,7 +258,7 @@ public boolean isFinished() { | |||
} | |||
|
|||
public boolean addListener(Consumer<Boolean> task) { | |||
if (EXECUTOR.getQueue().size() > Config.profile_process_blocking_queue_size) { | |||
if (EXECUTOR.getQueue().remainingCapacity() <= 0) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so only 'corePoolSize' threads will be created? because when queue is full ,we change to sync profile mode
Why I'm doing:
Async profile
uses a background threadpool to process profile, but it's configuration doesn't make sense for some intensive workload.thread_num
: default is 2queue_size
:thread_num * 128
, too largeFor a typical workload:
100 / (1/0.2)=20
threads to catchup the query profileWhat I'm doing:
Change the threadpool of
profile-worker
:thread_num
: max ascpu_cores*0.75
queue_size
: max asthread_num*4
For the above workload:
Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: