Arun sqc interop #5

arunjose696 · 2024-07-12T09:31:02Z

What do these changes do?

Adds interoperablity between different query compiler by introducing a decorator function to do type casting of all query compiler arguments to the type of left operand

result=qc_native.op(pandas_qc) # This would result in result being of type native_qc
result=pandas_qc.op(qc_native) # This would result in result being of type pandas_qc

The tests for interoperablity have been added by modifying tests in modin/tests/pandas/dataframe/ folder such that operations would be performed across dataframes with different query compilers.

first commit message and PR title follow format outlined here

NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.
passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
signed commit with git commit -s
Resolves #?
tests added and passing
module layout described at docs/development/architecture.rst is up-to-date

YarShev · 2024-07-12T09:57:43Z

.github/workflows/ci.yml

@@ -697,6 +697,16 @@ jobs:
      - run: python -m pytest modin/tests/pandas/dataframe/test_reduce.py
      - run: python -m pytest modin/tests/pandas/dataframe/test_udf.py
      - run: python -m pytest modin/tests/pandas/dataframe/test_window.py
+      - run: python -m pytest modin/tests/pandas/interop/test_binary.py


Suggested change

- run: python -m pytest modin/tests/pandas/interop/test_binary.py

- run: python -m pytest modin/tests/native_df_mode/pandas/interop/test_binary.py

I envision we could implement a native df mode for polars so let's change the test directory stucture a bit.

YarShev · 2024-07-12T09:59:00Z

modin/core/storage_formats/pandas/query_compiler_validator.py

+"""
+Module contains ``QueryCompilerTypeCaster`` class.
+
+``QueryCompilerTypeCaster`` is used for va.


Extend the docstring?

modin/core/storage_formats/pandas/query_compiler_validator.py

YarShev · 2024-07-12T11:50:10Z

modin/core/storage_formats/pandas/native_query_compiler.py

@@ -585,7 +586,6 @@ class NativeQueryCompiler(BaseQueryCompiler):
    _shape_hint: Optional[str]

    def __init__(self, pandas_frame, shape_hint: Optional[str] = None):
-        assert NativeDataframeMode.get() == "Pandas"


Why removed?

Now constructor of NativeQueryCompiler can be also called when NativeDataframeMode.get() is default, Irrespective of NativeDataframeMode when we call this constructor in operations where the left operand is a NativeQueryCompiler we construct a new query compiler of type NativeQueryCompiler

YarShev · 2024-07-12T11:52:45Z

modin/core/storage_formats/pandas/query_compiler_validator.py

+Fn = TypeVar("Fn", bound=Any)
+
+
+class QueryCompilerTypeCaster:


Suggested change

class QueryCompilerTypeCaster:

class QueryCompilerCaster:

YarShev · 2024-07-12T11:53:15Z

modin/core/storage_formats/pandas/query_compiler_validator.py

+    return arguments
+
+
+def apply_argument_casting():


Suggested change

def apply_argument_casting():

def apply_argument_cast():

YarShev · 2024-07-12T11:59:45Z

modin/core/storage_formats/pandas/query_compiler_validator.py

+        A decorator function.
+    """
+
+    def decorator(obj: Fn) -> Fn:


We should probably use functools.wrap(obj) to keep properties of a function correct.

Now added functools to cast_args the function returned from this decorator, from what I understand functools wrapping is required only for the wrapper method returned from the decorator and the decorator by itself does not need functools.wraps()

YarShev · 2024-07-12T11:59:57Z

modin/core/storage_formats/pandas/query_compiler_validator.py

+        elif isinstance(obj, staticmethod):
+            return staticmethod(decorator(obj.__func__))
+
+        def cast_args(*args: Tuple, **kwargs: Dict) -> Any:


YarShev · 2024-07-12T12:13:00Z

modin/pandas/dataframe.py

@@ -2993,9 +2993,8 @@ def _create_or_update_from_compiler(
        DataFrame or None
            None if update was done, ``DataFrame`` otherwise.
        """
-        assert (
-            isinstance(new_query_compiler, type(self._query_compiler))


Why removed?

new_query_compiler can also be an instance of a different query compiler(Nativequerycompiler or PandasQueryCompiler),

Eg during operations like insert in the _create_or_update_from_compiler the constructor gets called directly eg ,

Thus it would be normal for the cases where constructor may return a new query compiler for a case where user changes the query compiler mode between creating data_frame and insert operation

YarShev · 2024-07-12T12:13:31Z

modin/core/storage_formats/pandas/query_compiler_validator.py

+            current_qc = None
+            if isinstance(args[0], BaseQueryCompiler):
+                current_qc = args[0]
+
+            if current_qc:
+                args = cast_nested_args_to_current_qc_type(args, current_qc)


Suggested change

current_qc = None

if isinstance(args[0], BaseQueryCompiler):

current_qc = args[0]

if current_qc:

args = cast_nested_args_to_current_qc_type(args, current_qc)

if isinstance(args[0], BaseQueryCompiler):

current_qc = args[0]

args = cast_nested_args_to_current_qc_type(args, current_qc)

What about kwargs?

YarShev · 2024-07-12T12:16:32Z

modin/core/storage_formats/pandas/query_compiler_validator.py

+    def decorator(obj: Fn) -> Fn:
+        """Decorate function or class to cast all arguments that are query comilers to the current query compiler"""
+        if isinstance(obj, type):
+            seen: Dict[Any, Any] = {}


Why do we need this dict? Is there a case when we may not get into except KeyError branch?

The idea was to handle cases where for same function different aliases exists in basequerycompiler, This is no longer required and no cases in both the query compilers that would go to except so removing this.

Signed-off-by: Wei Chen <wei@runllm.com>

* FEAT-modin-project#7331: Initial Polars API This commit adds a polars namespace to Modin, and the DataFrame and Series objects and their respective APIs. This doesn't include error handling and is still missing several polars features: * LazyFrame * Expressions * String, Temporal, Struct, and other Series accessors * Several parameters * Operators that we don't have query compiler methods for * e.g. sin, cos, tan, etc. Those will be handled in a future PR. Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * Lint Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * flake8 Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * isort Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * headers Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * forgot one Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * Add test Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * header Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * isort Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * Add to CI Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * fix name Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * Update modin/polars/base.py Co-authored-by: Mahesh Vashishtha <mahesh.vashishtha@snowflake.com> * address comments Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * polars 1 Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * Update for polars 1.x and fix some hacks Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * Remove hax Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * Black Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * Address comments Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * Lint Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> * Address comment Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> --------- Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com> Co-authored-by: Devin Petersohn <devin.petersohn@snowflake.com> Co-authored-by: Mahesh Vashishtha <mahesh.vashishtha@snowflake.com>

…er (modin-project#7348) Signed-off-by: Jonathan Shi <jhshi07@gmail.com>

…odin-project#7352) Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>

…Dataset. (modin-project#7353) Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com>

…n-project#7354) Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com>

…er (modin-project#7356) Signed-off-by: arunjose696 <arunjose696@gmail.com>

…odin-project#7358) Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>

…amic partitioning (modin-project#7369) Signed-off-by: Kirill Suvorov <kirill.suvorov@intel.com> Co-authored-by: Anatoly Myachev <anatoliimyachev@mail.com>

…ervice, pin to 5.0.13 (modin-project#7374) Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

Co-authored-by: Igoshev, Iaroslav <iaroslav.igoshev@intel.com> Signed-off-by: arunjose696 <arunjose696@gmail.com>

…ply` (modin-project#7338) Signed-off-by: Kirill Suvorov <kirill.suvorov@intel.com>

Signed-off-by: arunjose696 <arunjose696@gmail.com>

Signed-off-by: Igoshev, Iaroslav <iaroslav.igoshev@intel.com>

…rialized_dtypes to query compiler layer as in the code in multiple places the methods of private _modin_frame were used

Signed-off-by: Igoshev, Iaroslav <iaroslav.igoshev@intel.com>

…fferent query compilers

Co-authored-by: Iaroslav Igoshev <Poolliver868@mail.ru>

Signed-off-by: arunjose696 <arunjose696@gmail.com>

Co-authored-by: Iaroslav Igoshev <Poolliver868@mail.ru> Co-authored-by: Anatoly Myachev <anatoliimyachev@mail.com>

Signed-off-by: arunjose696 <arunjose696@gmail.com>

arunjose696 force-pushed the arun-sqc-interop branch from 6594669 to 741b6d0 Compare July 12, 2024 09:42

YarShev reviewed Jul 12, 2024

View reviewed changes

ZhipengXue97 and others added 9 commits July 15, 2024 09:57

FIX-#0000: Fix type hint (modin-project#7343)

4815bc3

DOCS-#0000: Update RunLLM Ask AI widget script path (modin-project#7345)

f5f9ae9

Signed-off-by: Wei Chen <wei@runllm.com>

FEAT-modin-project#7340: Add more granular lazy flags to query compil…

a40cef7

…er (modin-project#7348) Signed-off-by: Jonathan Shi <jhshi07@gmail.com>

FIX-modin-project#7351: Add ipython method calls to non-lookup list (m…

621f49e

…odin-project#7352) Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>

FIX-modin-project#7134: Use a separate docstring class for BasePandas…

24018db

…Dataset. (modin-project#7353) Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com>

FIX-modin-project#7113: Fix docstring overrides for subclasses. (modi…

22ed4d8

…n-project#7354) Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com>

FIX-modin-project#7355: Cpu count would be set incorrectly on a clust…

6dce30e

…er (modin-project#7356) Signed-off-by: arunjose696 <arunjose696@gmail.com>

FIX-modin-project#7357: Fix NoAttributeError on DataFrame.copy (m…

b236b76

…odin-project#7358) Signed-off-by: Devin Petersohn <devin.petersohn@snowflake.com>

arunjose696 force-pushed the arun-sqc-interop branch 3 times, most recently from f788a20 to 68a51ef Compare August 14, 2024 16:18

Retribution98 and others added 13 commits August 19, 2024 13:56

FEAT-modin-project#7368: Add a new environment variable for using dyn…

05e5c48

…amic partitioning (modin-project#7369) Signed-off-by: Kirill Suvorov <kirill.suvorov@intel.com> Co-authored-by: Anatoly Myachev <anatoliimyachev@mail.com>

FIX-modin-project#7373: Try a previous version of motoserver/moto s…

8fc230a

…ervice, pin to 5.0.13 (modin-project#7374) Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

FEAT-modin-project#4605: Add native query compiler (modin-project#7259)

da01571

Co-authored-by: Igoshev, Iaroslav <iaroslav.igoshev@intel.com> Signed-off-by: arunjose696 <arunjose696@gmail.com>

FEAT-modin-project#7337: Using dynamic partitionning in `broadcast_ap…

8249915

…ply` (modin-project#7338) Signed-off-by: Kirill Suvorov <kirill.suvorov@intel.com>

FEAT-modin-project#4605: Add small query compiler

ba48d29

fixing tests

40e1cd4

removing additional parameter from try_cast_to_pandas

a574172

Signed-off-by: arunjose696 <arunjose696@gmail.com>

test_iter passing

6f9795b

fixing isin unique and clip

07d8d3a

Signed-off-by: arunjose696 <arunjose696@gmail.com>

Enable test_default.py and test_join_sort.py

ac75854

Signed-off-by: Igoshev, Iaroslav <iaroslav.igoshev@intel.com>

fixed test_map_metadata by adding set_frame_dtypes_cache and has_mate…

52b3136

…rialized_dtypes to query compiler layer as in the code in multiple places the methods of private _modin_frame were used

Fix test_dot

4aed823

Signed-off-by: Igoshev, Iaroslav <iaroslav.igoshev@intel.com>

test_udf passing

7149ba3

arunjose696 and others added 9 commits August 26, 2024 16:37

FEAT-modin-project#7308: Interoperability between DataFrames using di…

4e24de2

…fferent query compilers

Started tests for interop

2308295

modified test_binary and test_default

d240ac8

Started adding testindexing

a4c1697

fixing envvars repeatation

89dabf9

adding tests

b212c3b

PR comments and tests

90ba1a4

Update modin/core/storage_formats/pandas/query_compiler_validator.py

e9060a7

Co-authored-by: Iaroslav Igoshev <Poolliver868@mail.ru>

PR comments and tests

60f9512

arunjose696 force-pushed the arun-sqc-interop branch 5 times, most recently from cd70589 to 9af55f2 Compare August 26, 2024 15:29

Doc updates

80ce01c

Signed-off-by: arunjose696 <arunjose696@gmail.com>

arunjose696 force-pushed the arun-sqc-interop branch from 9af55f2 to 80ce01c Compare August 26, 2024 17:11

arunjose696 and others added 2 commits August 28, 2024 10:55

PR comments

6a068dc

Apply suggestions from code review

cac89d2

Co-authored-by: Iaroslav Igoshev <Poolliver868@mail.ru> Co-authored-by: Anatoly Myachev <anatoliimyachev@mail.com>

arunjose696 force-pushed the arun-sqc-interop branch 2 times, most recently from 94e8e48 to af0a7ba Compare August 28, 2024 10:10

PR changes

8e8ec46

Signed-off-by: arunjose696 <arunjose696@gmail.com>

arunjose696 force-pushed the arun-sqc-interop branch 4 times, most recently from 04b7f60 to 03d137c Compare August 29, 2024 07:15

creating function for create_test_df_in_defined_mode

c1b0942

Signed-off-by: arunjose696 <arunjose696@gmail.com>

arunjose696 force-pushed the arun-sqc-interop branch from 03d137c to c1b0942 Compare August 29, 2024 08:25

creating function for create_test_df_in_defined_mode

f488872

Signed-off-by: arunjose696 <arunjose696@gmail.com>

arunjose696 force-pushed the arun-sqc-interop branch 2 times, most recently from 136bf1e to f488872 Compare September 2, 2024 08:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Arun sqc interop #5

Arun sqc interop #5

arunjose696 commented Jul 12, 2024 •

edited

Loading

YarShev Jul 12, 2024

arunjose696 Aug 14, 2024

YarShev Jul 12, 2024

arunjose696 Aug 14, 2024

YarShev Jul 12, 2024

arunjose696 Aug 14, 2024 •

edited

Loading

YarShev Jul 12, 2024

arunjose696 Aug 14, 2024

YarShev Jul 12, 2024

arunjose696 Aug 14, 2024

YarShev Jul 12, 2024

arunjose696 Aug 14, 2024 •

edited

Loading

YarShev Jul 12, 2024

arunjose696 Aug 14, 2024

YarShev Jul 12, 2024

arunjose696 Aug 14, 2024

YarShev Jul 12, 2024

arunjose696 Aug 14, 2024

YarShev Jul 12, 2024

arunjose696 Aug 14, 2024

	- run: python -m pytest modin/tests/pandas/interop/test_binary.py
	- run: python -m pytest modin/tests/native_df_mode/pandas/interop/test_binary.py

Arun sqc interop #5

Are you sure you want to change the base?

Arun sqc interop #5

Conversation

arunjose696 commented Jul 12, 2024 • edited Loading

What do these changes do?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arunjose696 Aug 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arunjose696 Aug 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arunjose696 commented Jul 12, 2024 •

edited

Loading

arunjose696 Aug 14, 2024 •

edited

Loading

arunjose696 Aug 14, 2024 •

edited

Loading