-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[REVIEW] Speedup Connected Components #302
Open
VibhuJawa
wants to merge
12
commits into
NVIDIA:main
Choose a base branch
from
VibhuJawa:vjawa/speedup_cc_in_fuzzy_dedup
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
VibhuJawa
commented
Oct 15, 2024
VibhuJawa
commented
Oct 16, 2024
VibhuJawa
force-pushed
the
vjawa/speedup_cc_in_fuzzy_dedup
branch
from
October 16, 2024 02:35
f424de2
to
ea65bad
Compare
Signed-off-by: Vibhu Jawa <vibhujawa@gmail.com> Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vibhujawa@gmail.com> Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@login-eos02.eos.clusters.nvidia.com> Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
VibhuJawa
force-pushed
the
vjawa/speedup_cc_in_fuzzy_dedup
branch
from
October 16, 2024 02:46
ea65bad
to
5caa34a
Compare
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
VibhuJawa
commented
Oct 16, 2024
Signed-off-by: Vibhu Jawa <vibhujawa@gmail.com>
Signed-off-by: Vibhu Jawa <vibhujawa@gmail.com>
Signed-off-by: Vibhu Jawa <vibhujawa@gmail.com>
VibhuJawa
commented
Oct 17, 2024
VibhuJawa
changed the title
[WIP] Speedup Connected Components
[REVIEW] Speedup Connected Components
Oct 22, 2024
VibhuJawa
force-pushed
the
vjawa/speedup_cc_in_fuzzy_dedup
branch
from
October 22, 2024 22:42
df62a1f
to
8396237
Compare
praateekmahajan
approved these changes
Oct 23, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, would wait for @ayushdg to also TAL!
Great speedup 🎊
Signed-off-by: Vibhu Jawa <vibhujawa@gmail.com>
@ayushdg , Please take a look. Lets land this in soon. Have addressed all the changes |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request includes several changes to the
nemo_curator/modules/fuzzy_dedup.py
file, focusing on removing theconvert_str_ids
functionality, optimizing performance, and improving logging.The most important changes are:
Removal of
convert_str_ids
functionality:convert_str_ids
parameter and its associated logic from the__init__
method and other methods innemo_curator/modules/fuzzy_dedup.py
. [1] [2] [3] [4] [5] [6]This is done because now we have longstrings support in
cuDF
so we no longer need to convert string to int idsPerformance optimizations:
Decreased the block size for reading parquet files in
_write_dedup_parsed_id
[1] to a lesser value to allow scaling of drop_duplicates (which has a big memory overhead 16x+ ) to prevent OOMs, this will allow us to run CC at larger scales without requiring more hardware.Increased the chuck size in
_write_encoded_jaccard_pair
methods to improve merge performance, as with large base chunks, we have bigger transfers so the throughput of transfer is better on TCP [2]Updated the
_run_connected_components
method to initializeComms
withp2p=False
Merge Improvements:
ddf_id
column.Main: 22m 10s
PR: 444.85 s
Dask Profiles:
cc_profiles.zip
Logging improvements:
cc_workflow
method and end-to-end time logging for the workflow. [1] [2]Verify Equal Results:
376321911
376321911
Check same ids
376321911
CC: @ayushdg
Checklist