feat(ingest/dremio): Dremio Source Ingestion #11598

sagar-salvi-apptware · 2024-10-11T11:52:24Z

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_datahub_source_mapping.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py

mayurinehate

Can you please make use of report.warning and/or report.error for cases where user needs to be aware of error details - such as login failure due to missing PAT or incorrect password ?

…ar-salvi-apptware/datahub into feat/dremio-connector-source

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_aspects.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py

metadata-ingestion/docs/sources/dremio/dremio_recipe.yml

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_api.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_profiling.py

dremio-brock · 2024-11-06T03:39:21Z

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_sql_queries.py

+            SYS.PROJECT.HISTORY.JOBS
+        WHERE
+            STATUS = 'COMPLETED'
+            AND LENGTH(queried_datasets)>0


Length is not valid on an array field, use
AND ARRAY_SIZE(queried_datasets)>0
instead

feat: Dremio Source Ingestion

38e5946

github-actions bot added ingestion PR or Issue related to the ingestion of metadata docs Issues and Improvements to docs product PR or Issue related to the DataHub UI/UX devops PR or Issue related to DataHub backend & deployment labels Oct 11, 2024

vercel bot deployed to Preview October 11, 2024 12:11 View deployment

sagar-salvi-apptware added 4 commits October 11, 2024 18:48

refactor: improvement to metadata gathering

a5b8c45

fix: Update dremio_entity.py

4a81b19

fix: Update dremio_sql_queries.py

51c2443

refactor: Update dremio_source.py

84227b6

sagar-salvi-apptware changed the title ~~feat: Dremio Source Ingestion~~ feat(ingest/dremio): Dremio Source Ingestion Oct 11, 2024

vercel bot deployed to Preview October 11, 2024 13:42 View deployment

mayurinehate reviewed Oct 17, 2024

View reviewed changes

mayurinehate requested changes Oct 17, 2024

View reviewed changes

vercel bot deployed to Preview October 17, 2024 11:37 View deployment

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from 01ffbcd to bd1ad49 Compare October 17, 2024 16:29

sagar-salvi-apptware added 6 commits October 17, 2024 23:23

feat: Dremio Source Ingestion

e462be6

refactor: improvement to metadata gathering

703a3f8

fix: Update dremio_entity.py

4283cb6

fix: Update dremio_sql_queries.py

f897b9e

refactor: Update dremio_source.py

ac56777

test: add integration test for dremio

3ce57b9

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from c4fe5c5 to 084ce54 Compare October 17, 2024 17:53

vercel bot deployed to Preview October 17, 2024 18:21 View deployment

fix: added minor changes + fix testcase

e1ee817

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from 084ce54 to e1ee817 Compare October 18, 2024 06:32

vercel bot deployed to Preview October 18, 2024 06:54 View deployment

sagar-salvi-apptware and others added 3 commits October 21, 2024 13:41

fix: PR Comments

b23e003

Merge branch 'feat/dremio-connector-source' of https://github.com/sag…

7247ac1

…ar-salvi-apptware/datahub into feat/dremio-connector-source

switch to drill dialect

b9d7b8a

Merge branch 'master' into feat/dremio-connector-source

102b55f

vercel bot deployed to Preview October 28, 2024 12:26 View deployment

mayurinehate reviewed Oct 28, 2024

View reviewed changes

sagar-salvi-apptware added 2 commits October 28, 2024 20:06

fix: minor comments

8efa349

test: fix ci test

82988fd

mayurinehate self-requested a review October 28, 2024 14:58

vercel bot deployed to Preview October 28, 2024 15:12 View deployment

fix: added fixes for dremio cloud apis

7433b8b

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from ed91d1f to 7433b8b Compare October 28, 2024 16:35

vercel bot deployed to Preview October 28, 2024 17:13 View deployment

fix: added changes for profiling

dca1077

vercel bot deployed to Preview October 30, 2024 07:21 View deployment

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from a17e574 to 7231d64 Compare October 30, 2024 10:07

vercel bot had a problem deploying to Preview October 30, 2024 10:14 Failure

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from 7231d64 to 137b875 Compare October 30, 2024 18:35

test: updated tests for profiling

42e6ec2

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from 137b875 to 42e6ec2 Compare October 30, 2024 19:29

vercel bot had a problem deploying to Preview October 30, 2024 19:37 Failure

b-rockx reviewed Oct 31, 2024

View reviewed changes

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py Outdated Show resolved Hide resolved

b-rockx reviewed Oct 31, 2024

View reviewed changes

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py Outdated Show resolved Hide resolved

fix: cloud region

e27d542

vercel bot had a problem deploying to Preview October 31, 2024 07:25 Failure

mayurinehate approved these changes Nov 5, 2024

View reviewed changes

fix: minor changes

2f8d07c

vercel bot had a problem deploying to Preview November 5, 2024 11:50 Failure

mayurinehate added the merge-pending-ci A PR that has passed review and should be merged once CI is green. label Nov 5, 2024

Merge branch 'master' into feat/dremio-connector-source

ad405a9

vercel bot deployed to Preview November 5, 2024 15:10 View deployment

mayurinehate merged commit 5d17ecb into datahub-project:master Nov 5, 2024
77 checks passed

dremio-brock reviewed Nov 6, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ingest/dremio): Dremio Source Ingestion #11598

feat(ingest/dremio): Dremio Source Ingestion #11598

sagar-salvi-apptware commented Oct 11, 2024

mayurinehate left a comment

dremio-brock Nov 6, 2024

feat(ingest/dremio): Dremio Source Ingestion #11598

feat(ingest/dremio): Dremio Source Ingestion #11598

Conversation

sagar-salvi-apptware commented Oct 11, 2024

Checklist

mayurinehate left a comment

Choose a reason for hiding this comment

dremio-brock Nov 6, 2024

Choose a reason for hiding this comment