Upgrade spark dependency to 3.3.0 #824

WeichenXu123 · 2022-08-15T09:58:10Z

Upgrade spark dependency to 3.3.0

Closes #823

WeichenXu123 · 2022-08-15T10:09:43Z

CC @jsleight Could you help review and merge if it is good ?

WeichenXu123 · 2022-08-15T10:29:49Z

python/requirements.txt

@@ -1,7 +1,7 @@
 numpy>=1.8.2
 six>=1.10.0
 scipy>=0.13.0b1
-pandas>=0.18.1, <= 0.24.2
+pandas>=1.0.5


Spark 3.3 requires pandas>=1.0.5

WeichenXu123 · 2022-08-15T10:30:26Z

Makefile

-.PHONY: py36_test
-py36_test:
-	source scripts/scala_classpath_for_python.sh && make -C python py36_test


Spark 3.3 does not support python3.6

WeichenXu123 · 2022-08-15T12:45:15Z

project/Dependencies.scala

  lazy val awsSdkVersion = "1.11.1033"
  val tensorflowJavaVersion = "0.4.0" // Match Tensorflow 2.7.0 https://github.com/tensorflow/java/#tensorflow-version-support
  val xgboostVersion = "1.5.2"
-  val breezeVersion = "1.0"
+  val breezeVersion = "1.2"


Keep the same with spark 3.3 breeze dependency.

WeichenXu123 · 2022-08-15T12:45:52Z

project/Dependencies.scala

  val scalaTestVersion = "3.0.8"
  val junitVersion = "5.8.2"
  val akkaVersion = "2.6.14"
  val akkaHttpVersion = "10.2.4"
  val springBootVersion = "2.6.2"
  lazy val logbackVersion = "1.2.3"
  lazy val loggingVersion = "3.9.0"
-  lazy val slf4jVersion = "1.7.30"
+  lazy val slf4jVersion = "1.7.36"


Old log4j version has conflicts with spark 3.3 dependencies.

WeichenXu123 · 2022-08-15T12:46:51Z

python/mleap/sklearn/preprocessing/data.py

-            x = y.ix[:,0]
-            y = y.ix[:,1]
+            x = y.iloc[:,0]
+            y = y.iloc[:,1]


Because required pandas version >= 1.05, the .ix method is removed.

WeichenXu123 · 2022-08-15T13:11:48Z

CC @jsleight Could you help review and merge if it is good ?

The CI does not run on my last commit, could you help fix it ?

WeichenXu123 · 2022-08-18T11:58:55Z

CC @jsleight Thanks!

WeichenXu123 · 2022-08-19T01:00:03Z

CC @jsleight Thanks!

WeichenXu123 · 2022-09-01T03:09:47Z

CC @jsleight Thanks!

realknorke · 2022-09-08T07:34:42Z

Can please someone review this PR and maybe merge? Pretty please? Mleap is currently blocking Spark upgrades to latest stable…
Thank you! :D

ancasarb · 2022-09-14T12:04:05Z

@WeichenXu123 @realknorke sorry, we had run out of credits for travis CI, can you please push a dummy commit to trigger the build now? will merge it once it's green.

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

WeichenXu123 · 2022-09-15T11:43:14Z

python/mleap/sklearn/preprocessing/data.py

+            return pd.DataFrame(np.add(x, y), columns=[self.output_features])
        elif self.transform_type == 'sub':
-            return pd.DataFrame(np.subtract(x, y))
+            return pd.DataFrame(np.subtract(x, y), columns=[self.output_features])
        elif self.transform_type == 'mul':
-            return pd.DataFrame(np.multiply(x, y))
+            return pd.DataFrame(np.multiply(x, y), columns=[self.output_features])
        elif self.transform_type == 'div':
-            return pd.DataFrame(np.divide(x, y))
+            return pd.DataFrame(np.divide(x, y), columns=[self.output_features])
        elif self.transform_type == 'rem':
-            return pd.DataFrame(np.remainder(x, y))
+            return pd.DataFrame(np.remainder(x, y), columns=[self.output_features])
        elif self.transform_type == 'pow':
-            return pd.DataFrame(x**y)
+            return pd.DataFrame(x**y, columns=[self.output_features])


Note: A fix.

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

WeichenXu123 · 2022-09-15T12:32:35Z

python/tests/sklearn/preprocessing/data_test.py

-        assert_frame_equal(res_a, res_b)
+
+        # TODO: Deserialization on output_features has some issue. fix this.
+        # assert_frame_equal(res_a, res_b)


@ancasarb

Could you help fixing this ?

This is an existing bug but previous test does not cover it.

But this is not related to this PR.

can you make a github issue for this so we don't forget about it?

Filed ticket: #830

WeichenXu123 · 2022-09-16T12:45:39Z

CC @ancasarb PR is ready for merging :)

jsleight · 2022-09-22T19:53:06Z

python/tests/sklearn/preprocessing/data_test.py

-        assert_frame_equal(res_a, res_b)
+
+        # TODO: Deserialization on output_features has some issue. fix this.
+        # assert_frame_equal(res_a, res_b)


can you make a github issue for this so we don't forget about it?

jsleight · 2022-09-22T20:01:15Z

So before we merge this do we want to do one last mleap release on spark 3.2? There have been a reasonable number of bug fixes since last release and we might want to ship those out before requiring folks to do the spark 3.3 upgrade? cc @ancasarb

WeichenXu123 · 2022-09-29T13:38:13Z

CC @jsleight Can we merge now :) ?

jsleight · 2022-09-29T15:17:46Z

Personally I would like to release all the 3.2.x fixes before this upgrade, but I think only @ancasarb has the release keys.

WeichenXu123 · 2022-09-30T03:16:20Z

OK fine. Let's wait. CC @ancasarb

WeichenXu123 · 2022-10-27T11:21:24Z

CC @jsleight @ancasarb Any progress ? Thank you.

WeichenXu123 · 2022-11-04T12:42:56Z

@jsleight Thank you!

realknorke · 2023-02-08T12:04:08Z

This topic (Spark3.3) is getting more and more urgent. Would it be possible to at least create a release candidate for the next mleap release?

WeichenXu123 · 2023-02-08T14:20:34Z

@jsleight Can we release it for spark 3.3 ? :)

jsleight · 2023-02-08T15:23:08Z

releasing a new mleap version sgtm. I might have time to do this next week.

realknorke · 2023-02-08T16:35:15Z

Thank you very much, guys! :D

realknorke · 2023-02-23T10:08:07Z

@jsleight Can you please release a new version w/ Spark 3.3.2 support and I'll buy you a beer! ;)

jsleight · 2023-02-24T19:31:58Z

Apologies for the delay. I just finished releasing mleap v0.22.0 which is on spark 3.3.0. See the release notes for the full changelong. You should see the relevant artifacts on sonatype and pypi. Cheers!

init

8221606

WeichenXu123 mentioned this pull request Aug 15, 2022

Bump pyspark from 3.2.0 to 3.2.2 in /python #823

Closed

update

edcbb50

WeichenXu123 commented Aug 15, 2022

View reviewed changes

WeichenXu123 added 6 commits August 15, 2022 18:53

log warn

7493b35

spark log level warn

3badb5b

update

abd84c6

update

57d4e32

fix-slf4-err

2714f95

fix python test

82cca57

WeichenXu123 commented Aug 15, 2022

View reviewed changes

clean

f9c4fa8

WeichenXu123 requested a review from jsleight August 15, 2022 13:11

update

031c8c8

WeichenXu123 added 2 commits September 15, 2022 19:41

fix MathBinary

b1be5ff

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

Merge branch 'master' into upgrade-spark-3.3

4c30169

WeichenXu123 force-pushed the upgrade-spark-3.3 branch from 4bb9465 to 4c30169 Compare September 15, 2022 11:42

WeichenXu123 commented Sep 15, 2022

View reviewed changes

update

f80ced7

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

WeichenXu123 commented Sep 15, 2022

View reviewed changes

jsleight approved these changes Sep 22, 2022

View reviewed changes

Merge branch 'master' into upgrade-spark-3.3

a84b715

WeichenXu123 mentioned this pull request Sep 23, 2022

Fix MathBinary bug #830

Open

Merge branch 'master' into upgrade-spark-3.3

d18df17

jsleight merged commit aa93bab into combust:master Nov 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade spark dependency to 3.3.0 #824

Upgrade spark dependency to 3.3.0 #824

WeichenXu123 commented Aug 15, 2022 •

edited

Loading

WeichenXu123 commented Aug 15, 2022

WeichenXu123 Aug 15, 2022

WeichenXu123 Aug 15, 2022

WeichenXu123 Aug 15, 2022

WeichenXu123 Aug 15, 2022 •

edited

Loading

WeichenXu123 Aug 15, 2022

WeichenXu123 commented Aug 15, 2022

WeichenXu123 commented Aug 18, 2022

WeichenXu123 commented Aug 19, 2022

WeichenXu123 commented Sep 1, 2022

realknorke commented Sep 8, 2022

ancasarb commented Sep 14, 2022

WeichenXu123 Sep 15, 2022

WeichenXu123 Sep 15, 2022

jsleight Sep 22, 2022

WeichenXu123 Sep 23, 2022

WeichenXu123 commented Sep 16, 2022

jsleight Sep 22, 2022

jsleight commented Sep 22, 2022

WeichenXu123 commented Sep 29, 2022

jsleight commented Sep 29, 2022

WeichenXu123 commented Sep 30, 2022 •

edited

Loading

WeichenXu123 commented Oct 27, 2022

WeichenXu123 commented Nov 4, 2022

realknorke commented Feb 8, 2023

WeichenXu123 commented Feb 8, 2023

jsleight commented Feb 8, 2023

realknorke commented Feb 8, 2023

realknorke commented Feb 23, 2023

jsleight commented Feb 24, 2023

Upgrade spark dependency to 3.3.0 #824

Upgrade spark dependency to 3.3.0 #824

Conversation

WeichenXu123 commented Aug 15, 2022 • edited Loading

WeichenXu123 commented Aug 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WeichenXu123 Aug 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WeichenXu123 commented Aug 15, 2022

WeichenXu123 commented Aug 18, 2022

WeichenXu123 commented Aug 19, 2022

WeichenXu123 commented Sep 1, 2022

realknorke commented Sep 8, 2022

ancasarb commented Sep 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WeichenXu123 commented Sep 16, 2022

Choose a reason for hiding this comment

jsleight commented Sep 22, 2022

WeichenXu123 commented Sep 29, 2022

jsleight commented Sep 29, 2022

WeichenXu123 commented Sep 30, 2022 • edited Loading

WeichenXu123 commented Oct 27, 2022

WeichenXu123 commented Nov 4, 2022

realknorke commented Feb 8, 2023

WeichenXu123 commented Feb 8, 2023

jsleight commented Feb 8, 2023

realknorke commented Feb 8, 2023

realknorke commented Feb 23, 2023

jsleight commented Feb 24, 2023

WeichenXu123 commented Aug 15, 2022 •

edited

Loading

WeichenXu123 Aug 15, 2022 •

edited

Loading

WeichenXu123 commented Sep 30, 2022 •

edited

Loading