You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Our extension type system allows us to model any type that can be represented by another DType plus some opaque metadata.
We currently only have one Encoding, vortex.extension, that is reused for all extension-typed arrays. This falls short for the purposes of encoding extension types that we want to decode into specific Arrow types.
For example, Arrow has a GeoArrow extension that is quite popular. We should enable the import of GeoArrow datasets into vortex, and then be able to send GeoArrow back out.
This will require us to make a few decisions
How do we want to make a vtable/function registry available at runtime so that we can specialize into_arrow for ExtensionArray. This will require us to have some sort of global context that is writable by extension authors and accessible from compute functions
Arrow extension arrays like GeoArrow require additional metadata, which is attached to a Field schema and not just a DataType or ArrayRef. I don't think there's currently a place in our API to return that information
The text was updated successfully, but these errors were encountered:
Our extension type system allows us to model any type that can be represented by another
DType
plus some opaque metadata.We currently only have one Encoding,
vortex.extension
, that is reused for all extension-typed arrays. This falls short for the purposes of encoding extension types that we want to decode into specific Arrow types.For example, Arrow has a GeoArrow extension that is quite popular. We should enable the import of GeoArrow datasets into vortex, and then be able to send GeoArrow back out.
This will require us to make a few decisions
into_arrow
forExtensionArray
. This will require us to have some sort of global context that is writable by extension authors and accessible from compute functionsField
schema and not just aDataType
orArrayRef
. I don't think there's currently a place in our API to return that informationThe text was updated successfully, but these errors were encountered: