Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support table features / protocol versions #32

Open
MrPowers opened this issue Sep 19, 2023 · 1 comment
Open

Support table features / protocol versions #32

MrPowers opened this issue Sep 19, 2023 · 1 comment

Comments

@MrPowers
Copy link
Collaborator

Document the table features / protocol version that are supported by this project for each release.

If a Delta table has a table feature enabled that deltatorch does not support (like deletion vectors), then deltatorch should error out.

@wolliq
Copy link

wolliq commented Mar 1, 2024

Hi @MrPowers,

I have an issue reading delta tables using deltatorch when the lib wawnts to build the PyArrow dataset out of the delta table.
As latest Databricks runtimes create delta tables with high feature level (3, 7) and those are different from the ones I see into the code base:

MAX_SUPPORTED_READER_VERSION = 1
MAX_SUPPORTED_WRITER_VERSION = 2

the execution results in error:
DeltaProtocolError: The table's minimum reader version is 3 but deltalake only supports up to version 1.
coming from the code base here:

if self.protocol().min_reader_version > MAX_SUPPORTED_READER_VERSION:
    raise DeltaProtocolError(
        f"The table's minimum reader version is {self.protocol().min_reader_version} "
        f"but deltalake only supports up to version {MAX_SUPPORTED_READER_VERSION}."
    )

What can be done to overcome this error? Why is it necessary that deltatorch keeps the protocol feature level to (1,2) ?

Thx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants