Added example notebook for translation with ct2 model. #262

uahmed93 · 2024-09-25T19:11:12Z

Description

This PR is adding a translation example to hindi languge via ct2 model.
Issue here
This example depends on CrossFit's PR

Checklist

I am familiar with the Contributing Guide.
New or Existing tests cover these changes.
The documentation is up to date with these changes.

Signed-off-by: Ahmed Umair <ahmedumair.1993@gmail.com>

VibhuJawa

Left a initial review, thanks for working on this tutorial.

tutorials/translation/ct2_hindi_translation.ipynb

VibhuJawa · 2024-10-02T16:49:45Z

tutorials/translation/ct2_hindi_translation.ipynb

+      "/home/uahmed/Desktop/NeMo-Curator/.ndc/lib/python3.10/site-packages/sklearn/base.py:376: InconsistentVersionWarning: Trying to unpickle estimator LinearRegression from version 1.4.2 when using version 1.5.1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:\n",
+      "https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations\n",
+      "  warnings.warn(\n",
+      "GPU: 0, Part: 0:   0%|          | 0/10 [05:33<?, ?it/s]1.74s/it]\n",
+      "GPU: 0, Part: 0: 100%|██████████| 10/10 [00:17<00:00,  1.77s/it]\n"
+     ]
+    },


Given its a tutorial, we should show the output that writing it out, maybe try doing both .

VibhuJawa · 2024-10-02T16:53:39Z

tutorials/translation/ct2_hindi_translation.ipynb

+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ[\"DASK_DATAFRAME__QUERY_PLANNING\"] = \"False\"\n",


You can remove this, no need to add it now.

Yep, this should be removed.

tutorials/translation/ct2_hindi_translation.ipynb

VibhuJawa · 2024-10-02T16:57:13Z

tutorials/translation/ct2_hindi_translation.ipynb

+   "metadata": {},
+   "source": [
+    "# Indic Translation\n",
+    "This notebook demonstrate an example use of nemo-curator for Indic language generation via translation from English language. This workflow is accelarated by CrossFit, a library that leverages intellegent batching and RAPIDS to accelerate the offline inference on large datasets. \n",


Would suggest mentioning that we support large scale translation using multi node multi gpu setup

VibhuJawa · 2024-10-02T16:58:11Z

tutorials/translation/ct2_hindi_translation.ipynb

+    "# Indic Translation\n",
+    "This notebook demonstrate an example use of nemo-curator for Indic language generation via translation from English language. This workflow is accelarated by CrossFit, a library that leverages intellegent batching and RAPIDS to accelerate the offline inference on large datasets. \n",
+    "This example uses ctransalte2 model from [here](https://indictrans2-public.objectstore.e2enetworks.net/it2_preprint_ckpts/en-indic-preprint.zip), taken from IndicTrans2 github repo, [here](https://github.com/AI4Bharat/IndicTrans2/tree/main?tab=readme-ov-file#multilingual-translation-models)"
+   ]


Add a clear demarkation of import section here

tutorials/translation/ct2_hindi_translation.ipynb

VibhuJawa · 2024-10-02T17:00:00Z

tutorials/translation/ct2_hindi_translation.ipynb

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Define output directory\n"


Suggested change

"## Define output directory\n"

"##### Define output directory\n"

VibhuJawa · 2024-10-02T17:01:42Z

tutorials/translation/ct2_hindi_translation.ipynb

+   ]
+  },
+  {
+   "cell_type": "markdown",


Add a table of contents here.

See https://github.com/NVIDIA/NeMo-Curator/blob/cef1c3408684e9ce96e8125f041378407daf0a7b/tutorials/image-curation/image-curation.ipynb example for input

tutorials/translation/ct2_hindi_translation.ipynb

Signed-off-by: Ahmed Umair <ahmedumair.1993@gmail.com>

uahmed93 · 2024-10-08T10:21:14Z

Hi @VibhuJawa / @sarahyurick ,
can you please merge this MR ?

sarahyurick

I can leave a more detailed review for focusing on grammar nits, but for now can you please implement the code changes from my review, then rerun the entire notebook (executing from start to bottom and in order)?

sarahyurick · 2024-10-10T20:56:25Z

tutorials/translation/ct2_hindi_translation.ipynb

+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ[\"DASK_DATAFRAME__QUERY_PLANNING\"] = \"False\"\n",


Yep, this should be removed.

sarahyurick · 2024-10-10T21:02:25Z

tutorials/translation/ct2_hindi_translation.ipynb

+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@dataclass\n",


How about removing this entire cell and creating a new script in https://github.com/NVIDIA/NeMo-Curator/tree/main/nemo_curator/classifiers?

We cant really have another classifier just yet, we have to have this as a tutorial as we dont want to support this globally just yet. We can probably create a utils file here to make this cleaner though.

Ah ok I see, using a utils file sounds good to me.

sarahyurick · 2024-10-10T21:03:03Z

tutorials/translation/ct2_hindi_translation.ipynb

+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class IndicTranslation(DistributedDataClassifier):\n",


Same comment as above, why don't we move all of this logic into https://github.com/NVIDIA/NeMo-Curator/tree/main/nemo_curator/classifiers?

uahmed93 mentioned this pull request Sep 25, 2024

Translation example with ctranslate2's Translator. #246

Open

uahmed93 force-pushed the uahmed/ct2_translation_example_notebook branch 2 times, most recently from 1b7e864 to 5ecd4fe Compare September 27, 2024 07:06

Added example notebook for translation with ct2 model.

cef1c34

Signed-off-by: Ahmed Umair <ahmedumair.1993@gmail.com>

uahmed93 force-pushed the uahmed/ct2_translation_example_notebook branch from 5ecd4fe to cef1c34 Compare October 2, 2024 13:52

VibhuJawa requested changes Oct 2, 2024

View reviewed changes

Restructured according to comments.

548f008

Signed-off-by: Ahmed Umair <ahmedumair.1993@gmail.com>

sarahyurick added the documentation Improvements or additions to documentation label Oct 7, 2024

sarahyurick requested changes Oct 10, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added example notebook for translation with ct2 model. #262

Added example notebook for translation with ct2 model. #262

uahmed93 commented Sep 25, 2024 •

edited

Loading

VibhuJawa left a comment

VibhuJawa Oct 2, 2024

VibhuJawa Oct 2, 2024

sarahyurick Oct 10, 2024

VibhuJawa Oct 2, 2024

VibhuJawa Oct 2, 2024

VibhuJawa Oct 2, 2024

VibhuJawa Oct 2, 2024

uahmed93 commented Oct 8, 2024

sarahyurick left a comment

sarahyurick Oct 10, 2024

sarahyurick Oct 10, 2024

VibhuJawa Oct 10, 2024 •

edited

Loading

sarahyurick Oct 10, 2024

sarahyurick Oct 10, 2024

	"## Define output directory\n"
	"##### Define output directory\n"

Added example notebook for translation with ct2 model. #262

Are you sure you want to change the base?

Added example notebook for translation with ct2 model. #262

Conversation

uahmed93 commented Sep 25, 2024 • edited Loading

Description

Checklist

VibhuJawa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uahmed93 commented Oct 8, 2024

sarahyurick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VibhuJawa Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uahmed93 commented Sep 25, 2024 •

edited

Loading

VibhuJawa Oct 10, 2024 •

edited

Loading