Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v0.0.2 partitioned tables have some strange partition values #35

Open
MrPowers opened this issue May 9, 2023 · 2 comments
Open

v0.0.2 partitioned tables have some strange partition values #35

MrPowers opened this issue May 9, 2023 · 2 comments

Comments

@MrPowers
Copy link
Collaborator

MrPowers commented May 9, 2023

The basic_partitioned reference table has a _HIVE_DEFAULT_PARTITION_:

Screenshot 2023-05-09 at 7 03 03 AM

The multi_partitioned also has some strange partition values:

Screenshot 2023-05-09 at 7 14 19 AM

Can we remove these partitions and also the .crc files?

@MrPowers
Copy link
Collaborator Author

MrPowers commented May 9, 2023

Upon further investigation, it seems like these values were added intentionally.

Here’s my proposal for next steps:

  • update basic_partitioned to not include any null partition values
  • update multi_partitioned to not include any null partition values
  • Add basic_partitioned_with_null (which will be the existing basic_partitioned just renamed)
  • Add multi_partitioned_with_null (just the existing multi_partitioned renamed)

AND

  • Just figure out what configurations need to be set so crc files aren't written

Open to thoughts.

@wjones127
Copy link
Collaborator

The whole point of DAT is to demonstrate cases that the original implementors might not have thought of.

There is one partition value that I think it's wrong, and that's the binary 👿 emoji. I think Spark is in the wrong for not percent-encoding that in the Path, and I'm unable to make a sensible way to read it in delta-rs. So I will remove that one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants