ARROW-238 Add Support for nested ObjectIDs in polars conversion #220
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
_arrow_to_polars currently has no support to cast extension types for nested fields. This prohibits ObjectIDs to be read in case they are in nested fields.
|
Thank you for you submission. It looks good to me. We are waiting on Polars to support ExtensionTypes, but in the meantime, I don't see why we wouldn't add this. I cannot recall why we commented out the list and struct cases before. Please give us a few days to review. Here is the link to the mongo-arrow task: https://jira.mongodb.org/browse/ARROW-202. It contains links to the Polars issues. |
|
Hi @sibbiii . I'm sorry for the delay. I've been very busy. Would you please add a couple tests of this new functionality? |
lazargugleta
force-pushed
the
patch-1
branch
2 times, most recently
from
14933f8
to
c771c0d
Compare
|
Hey @caseyclements , |
blink1073
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi,
_arrow_to_polars currently has no support to cast extension types for nested fields.
This prohibits ObjectIDs to be read in case they are in nested fields.
I could not manage the conversion with the original code,
but I found a way to using
arrow_table_without_extensions = arrow_table.cast(schema_without_extensions)to cast the schema of the whole table in one go.
The schema_without_extensions is created recursively from the old schema.
Support for lists is still to be added, should not be that hard, maybe I try tomorrow.
I am not an expert in apache arrow. My world is Pandas and Polars.
I have wrote some unit tests locally to test the code, but I do not feel confident that I have not overlooked
something, so please review carefully.
#219