Support time travel in Hudi connector #15084

albericgenius · 2022-11-17T16:08:29Z

Description

This PR only implement the incremental query for COW table.
Will discuss and implement the MOR table after MOR reading merged.
This PR depend on the TPCH modification, and the PR have not passed yet. so the build probably will be failed. :)

Additional context and related issues

https://issues.apache.org/jira/browse/HUDI-2692
https://issues.apache.org/jira/browse/HUDI-2693

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`15003 `)

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiConfig.java

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiErrorCode.java

...in/trino-hudi/src/main/java/io/trino/plugin/hudi/query/HudiReadOptimizedDirectoryLister.java

plugin/trino-hudi/src/test/java/io/trino/plugin/hudi/TestHudiCopyOnWriteConnectorTest.java

testing/trino-server-dev/etc/catalog/hudi.properties

testing/trino-testing/src/main/java/io/trino/testing/AbstractTestQueryFramework.java

testing/trino-testing/src/main/java/io/trino/testing/H2QueryRunner.java

ebyhr · 2022-11-18T00:31:32Z

Fix #15003

This isn't a time travel feature. Please take a look at Iceberg page in the issue.

plugin/trino-hudi/pom.xml

codope · 2022-11-18T09:29:45Z

Fix #15003

This isn't a time travel feature. Please take a look at Iceberg page in the issue.

@ebyhr Thanks for pointing to the commit. Time travel is just a fancy name for a versioned snapshot query. While we're using _hoodie_commit_time as a filter in this PR, AS OF semantics in SQL is much better suited for this feature. I couldn't find SELECT ... AS OF in Trino SQL docs. Is it only supported for Iceberg connector?

ebyhr · 2022-11-18T09:33:26Z

Is it only supported for Iceberg connector?

@codope Yes.

codope · 2022-11-18T09:38:19Z

Got it. I think this commit is more relevant for us 1c23c69 which adds the necessary connector metadata APIs to support versioned snapshot queries. If we implement those APIs in HudiMetadata, we can run SELECT ... AS OF queries through Hudi connector. cc @albericgenius

...ain/src/main/java/io/trino/sql/planner/iterative/rule/HudiPushTableVersionIntoTableScan.java

maddy2u · 2023-01-03T20:20:23Z

When are we expecting this to be merged ?

ebyhr · 2023-01-03T23:43:33Z

It depends on @albericgenius. I've waited that @albericgenius addresses comments.

albericgenius · 2023-01-10T02:51:32Z

It depends on @albericgenius. I've waited that @albericgenius addresses comments.

Got you, I will continue to work on this.

maddy2u · 2023-01-28T18:03:30Z

Hey @albericgenius - Any update on this?

albericgenius · 2023-02-09T03:48:36Z

Hey @albericgenius - Any update on this?

I updated a new revision, removed the Hudi specific code in the planner

@ebyhr @electrum @codope Thanks for your coaching and time.

Addressed

ebyhr

Could you add "Time travel queries" section to hudi.rst likes Iceberg documentation? It would be nice to mention the way to find versions from existing Hudi tables.

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiTableHandle.java

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiUtil.java

plugin/trino-hudi/src/test/java/io/trino/plugin/hudi/BaseHudiConnectorTest.java

plugin/trino-hudi/src/test/java/io/trino/plugin/hudi/TestHudiSmokeTest.java

docs/src/main/sphinx/connector/hudi.rst

albericgenius · 2023-04-06T05:49:29Z

@ebyhr i used the UI merge tool to resolve conflict issue, after that there is CI error: PR requires a rebase. Found: 1 merge.commit. and the cassandra CI test is not relative to my modification.

ebyhr · 2023-04-06T05:52:53Z

Please rebase on master instead.

albericgenius · 2023-04-07T11:08:32Z

Please rebase on master instead.

Got you

nikoshet · 2023-04-10T11:10:34Z

I guess this is close to getting merged 👀

albericgenius · 2023-04-13T09:10:49Z

@ebyhr still need your help :)

ebyhr

Please rebase on master to resolve confclits.

ebyhr · 2023-05-07T23:41:22Z

docs/src/main/sphinx/connector/hudi.rst

@@ -220,6 +220,46 @@ The output of the query has the following columns:
    - ``varchar``
    - Current state of the instant

+Rolling back to a previous version


@mosabua Could you please review this docs?

ebyhr · 2023-05-07T23:43:54Z

...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java

+        String after = formatter.format(new Date());
+
+        try {
+            assertThat(onTrino().executeQuery("SELECT id, name FROM hudi.default." + tableName + " FOR VERSION AS OF " + before)).hasNoRows();


It would be better to get the version from $timeline table.

ebyhr · 2023-05-07T23:53:01Z

...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java

+            onHudi().executeQuery("ALTER TABLE default." + tableName + " DROP COLUMNS (new_col)");
+            onHudi().executeQuery("INSERT INTO default." + tableName + " VALUES (4, 'a4', 20, 1000)");
+            String afterDropColumn = formatter.format(new Date());
+            assertThat(onTrino().executeQuery("SELECT id, name FROM hudi.default." + tableName + " FOR VERSION AS OF " + afterDropColumn))


The assertions having FOR VERSION AS OF specifies the latest version. That's not a common scenario when using time travel. I would recommend verifying with the past version. Also, it should verify the table definition.

ebyhr · 2023-05-07T23:55:22Z

...auncher/src/main/java/io/trino/tests/product/launcher/env/environment/EnvSinglenodeHudi.java

        builder.addConnector(
                "hive",
                forHostPath(configDir.getPath("hive.properties")),
                CONTAINER_TRINO_ETC + "/catalog/hive.properties");
+


nit: Revert unrelated change.

ebyhr · 2023-05-07T23:55:55Z

...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java

+    @Test(groups = {HUDI, PROFILE_SPECIFIC_TESTS})
+    public void testTimeTravelQuery()
+    {
+        String tableName = "test_hudi_cow_select_session_props" + randomNameSuffix();


Why the table name contains select_session_props?

ebyhr · 2023-05-07T23:56:48Z

docs/src/main/sphinx/connector/hudi.rst

+internally used for providing the previous state of the table::
+
+   SELECT *
+   FROM example.testdb.customer_orders FOR TIMESTAMP AS OF TIMESTAMP '2022-03-23 09:59:29.803 Europe/Vienna'


Does this PR contain a test case for FOR TIMESTAMP AS OF TIMESTAMP?

ebyhr · 2023-05-08T00:01:23Z

...rino-product-tests/src/main/java/io/trino/tests/product/hudi/TestHudiSparkCompatibility.java

+        formatter.setTimeZone(TimeZone.getTimeZone(ZoneOffset.UTC));
+        String before = formatter.format(new Date());
+        onHudi().executeQuery("SET hoodie.schema.on.read.enable=true");
+        createNonPartitionedTable(tableName, COW_TABLE_TYPE);


Please add another test for MOR table type.

ravjot28 · 2023-06-12T14:43:15Z

@ebyhr @albericgenius I have also implemented the time travel feature in my forked repo and the branch was cut from 418 tag, should i raise the PR to incorporate the feature?

Please let me know if it sounds good i will raise the PR with the desired changes

tooptoop4 · 2023-06-22T21:06:14Z

needs rebase

jlucking2023 · 2023-11-20T20:55:35Z

any word on this getting released - we've built an EntityMatch feature into InsuranceLake which stores data in an an Entity Primary Table that uses Hudi. We'd like to be able to use Athena to do 'Time Travel'...

aviral-nayya · 2024-01-04T17:46:44Z

Is this getting any love anytime soon?

mosabua · 2024-01-11T23:30:27Z

👋 @albericgenius - this PR has become inactive. We hope you are still interested in working on it. Please let us know, and we can try to get reviewers to help with that.

We're working on closing out old and inactive PRs, so if you're too busy or this has too many merge conflicts to be worth picking back up, we'll be making another pass to close it out in a few weeks.

Also fyi @codope @brandyml

github-actions · 2024-09-04T17:06:33Z

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

mosabua · 2024-09-06T23:24:49Z

Closing this PR due to inactivity. Anyone interested please feel free to pick it back up and continue work on it here or in a new PR.

cla-bot bot added the cla-signed label Nov 17, 2022

ebyhr reviewed Nov 18, 2022

View reviewed changes

electrum reviewed Nov 18, 2022

View reviewed changes

plugin/trino-hudi/pom.xml Outdated Show resolved Hide resolved

albericgenius changed the title ~~Support HUDI incremental query~~ Support HUDI incremental query for COW table Nov 18, 2022

ebyhr assigned albericgenius Nov 25, 2022

albericgenius force-pushed the incremental_query branch 2 times, most recently from a6c49d2 to 87553d8 Compare December 1, 2022 08:18

ebyhr previously requested changes Dec 1, 2022

View reviewed changes

...ain/src/main/java/io/trino/sql/planner/iterative/rule/HudiPushTableVersionIntoTableScan.java Outdated Show resolved Hide resolved

albericgenius commented Dec 1, 2022

View reviewed changes

...ain/src/main/java/io/trino/sql/planner/iterative/rule/HudiPushTableVersionIntoTableScan.java Outdated Show resolved Hide resolved

...ain/src/main/java/io/trino/sql/planner/iterative/rule/HudiPushTableVersionIntoTableScan.java Outdated Show resolved Hide resolved

ebyhr changed the title ~~Support HUDI incremental query for COW table~~ Support time travel in Hudi connector Dec 6, 2022

tooptoop4 mentioned this pull request Jan 3, 2023

[SUPPORT] Hudi Time Travel from Athena apache/hudi#7594

Open

albericgenius force-pushed the incremental_query branch from 87553d8 to 4c6c0d6 Compare February 9, 2023 03:43

albericgenius force-pushed the incremental_query branch from 4c6c0d6 to 147db05 Compare February 9, 2023 08:50

albericgenius requested a review from electrum February 10, 2023 02:23

albericgenius force-pushed the incremental_query branch from 147db05 to 4b5412d Compare February 10, 2023 10:25

albericgenius requested a review from ebyhr February 11, 2023 01:08

ebyhr reviewed Feb 14, 2023

View reviewed changes

albericgenius force-pushed the incremental_query branch from 4b5412d to 719cd8a Compare February 16, 2023 02:24

ebyhr reviewed Feb 16, 2023

View reviewed changes

docs/src/main/sphinx/connector/hudi.rst Outdated Show resolved Hide resolved

github-actions bot added the docs label Feb 16, 2023

albericgenius force-pushed the incremental_query branch from 719cd8a to 7c7cac5 Compare February 17, 2023 06:48

github-actions bot added the tests:hive label Feb 17, 2023

albericgenius force-pushed the incremental_query branch from 7c7cac5 to 56975af Compare February 17, 2023 12:05

albericgenius requested a review from ebyhr February 20, 2023 06:30

ebyhr reviewed Feb 20, 2023

View reviewed changes

docs/src/main/sphinx/connector/hudi.rst Outdated Show resolved Hide resolved

albericgenius force-pushed the incremental_query branch from 56975af to e0a084c Compare March 9, 2023 06:37

albericgenius requested a review from ebyhr March 9, 2023 06:48

github-actions bot added the hudi Hudi connector label Mar 9, 2023

kokosing force-pushed the master branch from 3f05134 to 58d6356 Compare March 14, 2023 11:34

albericgenius force-pushed the incremental_query branch from e0a084c to da2b230 Compare March 23, 2023 13:57

albericgenius force-pushed the incremental_query branch from f13e1d7 to 980d11a Compare April 7, 2023 05:01

Support time travel in Hudi connector

350bf9b

albericgenius force-pushed the incremental_query branch from 980d11a to 350bf9b Compare April 26, 2023 05:51

ebyhr reviewed May 7, 2023

View reviewed changes

ebyhr reviewed May 8, 2023

View reviewed changes

findepi removed the tests:hive label Apr 18, 2024

github-actions bot added the stale label Sep 4, 2024

mosabua closed this Sep 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support time travel in Hudi connector #15084

Support time travel in Hudi connector #15084

albericgenius commented Nov 17, 2022 •

edited

Loading

ebyhr commented Nov 18, 2022

codope commented Nov 18, 2022

ebyhr commented Nov 18, 2022

codope commented Nov 18, 2022 •

edited

Loading

maddy2u commented Jan 3, 2023

ebyhr commented Jan 3, 2023

albericgenius commented Jan 10, 2023

maddy2u commented Jan 28, 2023

albericgenius commented Feb 9, 2023

ebyhr left a comment

albericgenius commented Apr 6, 2023

ebyhr commented Apr 6, 2023

albericgenius commented Apr 7, 2023

nikoshet commented Apr 10, 2023

albericgenius commented Apr 13, 2023

ebyhr left a comment

ebyhr May 7, 2023

ebyhr May 7, 2023

ebyhr May 7, 2023

ebyhr May 7, 2023

ebyhr May 7, 2023

ebyhr May 7, 2023

ebyhr May 8, 2023

ravjot28 commented Jun 12, 2023 •

edited

Loading

tooptoop4 commented Jun 22, 2023

jlucking2023 commented Nov 20, 2023

aviral-nayya commented Jan 4, 2024

mosabua commented Jan 11, 2024

github-actions bot commented Sep 4, 2024

mosabua commented Sep 6, 2024 •

edited

Loading

Support time travel in Hudi connector #15084

Support time travel in Hudi connector #15084

Conversation

albericgenius commented Nov 17, 2022 • edited Loading

Description

Additional context and related issues

Release notes

ebyhr commented Nov 18, 2022

codope commented Nov 18, 2022

ebyhr commented Nov 18, 2022

codope commented Nov 18, 2022 • edited Loading

maddy2u commented Jan 3, 2023

ebyhr commented Jan 3, 2023

albericgenius commented Jan 10, 2023

maddy2u commented Jan 28, 2023

albericgenius commented Feb 9, 2023

ebyhr left a comment

Choose a reason for hiding this comment

albericgenius commented Apr 6, 2023

ebyhr commented Apr 6, 2023

albericgenius commented Apr 7, 2023

nikoshet commented Apr 10, 2023

albericgenius commented Apr 13, 2023

ebyhr left a comment

Choose a reason for hiding this comment

ebyhr May 7, 2023

Choose a reason for hiding this comment

ebyhr May 7, 2023

Choose a reason for hiding this comment

ebyhr May 7, 2023

Choose a reason for hiding this comment

ebyhr May 7, 2023

Choose a reason for hiding this comment

ebyhr May 7, 2023

Choose a reason for hiding this comment

ebyhr May 7, 2023

Choose a reason for hiding this comment

ebyhr May 8, 2023

Choose a reason for hiding this comment

ravjot28 commented Jun 12, 2023 • edited Loading

tooptoop4 commented Jun 22, 2023

jlucking2023 commented Nov 20, 2023

aviral-nayya commented Jan 4, 2024

mosabua commented Jan 11, 2024

github-actions bot commented Sep 4, 2024

mosabua commented Sep 6, 2024 • edited Loading

albericgenius commented Nov 17, 2022 •

edited

Loading

codope commented Nov 18, 2022 •

edited

Loading

ravjot28 commented Jun 12, 2023 •

edited

Loading

mosabua commented Sep 6, 2024 •

edited

Loading