feat(datasets): add pandas.DeltaSharingDataset

[hugodscarvalho]

Overview

This PR introduces a new dataset called DeltaSharingDataset, designed to load data from Delta Sharing shared tables into Pandas DataFrames. Delta Sharing is an open protocol that allows organizations to securely exchange large datasets in real-time, independent of the computing platforms they use. The dataset supports read-only operations and provides a way to integrate Delta Sharing data into Kedro workflows for data analysis and processing.

• Documentation: https://github.com/delta-io/delta-sharing.

Features

• Protocol: The DeltaSharingDataset is built using the Delta Sharing open protocol, enabling secure real-time data exchange.
• Data Loading: It loads data into a Pandas DataFrame by leveraging the delta_sharing.load_as_pandas function, allowing for easy data manipulation and analysis within Kedro pipelines.
• Versioning: You can specify a particular version of the dataset or load the latest version by default.
• Row Limiting: Supports limiting the number of rows loaded for previewing or partial data loading.
• Delta Format Option: Optionally load data in Delta format by setting the use_delta_format argument.
• Profile Credentials: Access to Delta Sharing tables is handled via a credentials dictionary, where the path to the Delta Sharing profile must be provided.

Example Usage

YAML API:

my_delta_sharing_dataset:
  type: pandas.DeltaSharingDataset
  share: <share-name>
  schema: <schema-name>
  table: <table-name>
  credentials:
    profile_file: <profile-file-path>
  load_args:
    version: <version>
    limit: <limit>
    use_delta_format: <use_delta_format>

Python API:

from kedro_datasets import DeltaSharingDataset
import pandas as pd

credentials = {
    "profile_file": "conf/local/config.share"
}
load_args = {
    "version": 1,
    "limit": 10,
    "use_delta_format": True
}

dataset = DeltaSharingDataset(
    share="example_share",
    schema="example_schema",
    table="example_table",
    credentials=credentials,
    load_args=load_args
)
data = dataset.load()
print(data)

Key Configuration Parameters

• share: The Delta Sharing share name.
• schema: The schema name within the share.
• table: The table name to load data from.
• credentials.profile_file: Path to the Delta Sharing profile file.
• load_args.version: The version of the table snapshot to load. If not provided, the latest version is loaded.
• load_args.limit: Maximum number of rows to load. Useful for data previews.
• load_args.use_delta_format: Whether to use Delta format for loading data. Defaults to False.

Limitations

• No Save Support: The DeltaSharingDataset is read-only and does not support saving data back to Delta Sharing tables.

Impact

This new dataset offers a simple, cost-effective way to incorporate Delta Sharing data into Kedro projects. It is especially useful in environments where shared data is accessed frequently for analysis, enabling users to leverage Delta Sharing's protocol for data interoperability without the need for heavy compute resources.

Why Delta Sharing?

• Interoperability: Delta Sharing allows data sharing between platforms without locking into a specific infrastructure.
• Cost-Efficiency: With read-only access, it minimizes resource usage by separating data storage and compute resources.
• Security: Built on a secure, REST-based protocol for trusted data sharing.

By adding this dataset, users can connect to Delta Sharing shared tables and manage large datasets in Pandas for data science tasks, making Kedro more versatile in handling modern data-sharing use cases.

Future Improvements

• Enhanced dataset operations for Spark DataFrames.

[ankatiyar]

Hi @hugodscarvalho, thank you so much for this contribution! Leaving a minor comment.

Would you mind also adding this to the release notes and the docs API .rst file here - https://github.com/kedro-org/kedro-plugins/blob/main/kedro-datasets/docs/source/api/kedro_datasets_experimental.rst

[ankatiyar]

This should be DatasetError which can be imported from kedro.io.core as I see some other datasets where one of the operations are not supported do the same.

[hugodscarvalho]

Totally agree! I have updated the code to use DatasetError imported from kedro.io.core instead of NotImplementedError, following the pattern in other datasets. Thank you for your feedback!

[hugodscarvalho]

Additionally, added the requested information to the release notes and the API .rst file as you suggested. If there's anything else you need, feel free to let me know!

[noklam]

Love what Delta is doing in the Python space and thank you for contributing the dataset. While we don't require fully tested dataset for experimental dataset, is is possible to share a runnable example that can be copy & paste? It would really help and make the review process easier as I am not able to get it run so it's tricky to review the code. It's looking very good though!

[noklam]

Why version 1.1.1?

[hugodscarvalho]

I’ve removed the version pinning since it was only necessary for testing purposes.

[hugodscarvalho]

Thank you for the kind words! The direction that Delta is taking was the exact reason that inspired me to contribute to the project and combine these two amazing initiatives. Regarding the runnable example, I have shared it in a different comment mentioning it. Please let me know if there's anything else you need!

[noklam]

Is it possible to create an example that can run locally? or is it expected to connect to somewhere?

[hugodscarvalho]

Thank you for your feedback, really appreciate it.

For accessing the Delta Sharing Server, Delta Sharing provides a profile file that allows you to connect to a public example server. You can download the necessary credentials from this link. More information about the Delta Sharing project can be found here, their Github repository.

Regarding the runnable example, I’m not sure if you’d prefer it in the code or in the comments, but I’ll provide it here for convenience:

from kedro_datasets_experimental.pandas import DeltaSharingDataset

# Define the credentials for accessing the Delta Sharing profile
credentials = {
    "profile_file": "open-datasets.share"  # Path to the profile file downloaded from link above
}

# Load arguments specifying any additional options for loading data
load_args = {
    "limit": 10,  # Limit the number of rows loaded
}

# Create an instance of the DeltaSharingDataset with the specified parameters
dataset = DeltaSharingDataset(
    share="delta_sharing",  # Name of the share in Delta Sharing
    schema="default",       # Schema name in Delta Sharing
    table="nyctaxi_2019",  # Table name to load data from
    credentials=credentials,  # Pass the credentials for access
    load_args=load_args      # Pass the loading options
)

# Load the data into a Pandas DataFrame
data = dataset.load()
print(data)  # Display the loaded data

The public example share includes the following datasets if you'd like to test different examples:

• COVID_19_NYT
• boston-housing
• flight-asa_2008
• lending_club
• nyctaxi_2019
• nyctaxi_2019_part
• owid-covid-data

Please let me know if you need any further information or if you'd like me to incorporate this example directly into the code!

[ElenaKhaustova]

@hugodscarvalho, thank you for the contribution, it looks great!

Agree with the point about testing - it would be helpful for us to get some notes on how to test it locally if possible.

Happy to approve when opened nit comments are resolved 🙂