DRAFT: Extension types #451
Conversation
pitrou
force-pushed
the
extension_types
branch
from
8b0da72
to
042a207
Compare
| When a reader encounters an extension type in a Parquet schema, it should try | ||
| to match it by name to its known extension types. If it does not recognize | ||
| the extension type, then it should read it as the underlying physical type | ||
| and should not try to interpret the column's statistics. It may however |
There was a problem hiding this comment.
min/max statistics, others should be valid?
|
Generally seems reasonable to me. |
| When a reader encounters an extension type in a Parquet schema, it should try | ||
| to match it by name to its known extension types. If it does not recognize | ||
| the extension type, then it should read it as the underlying physical type | ||
| and should not try to interpret the column's statistics. It may however |
| * | ||
| * If the extension type is not parametric, then `serialization` is empty. | ||
| */ | ||
| struct ExtensionTypeDescription { |
There was a problem hiding this comment.
Why choosing a dedicated ExtensionTypeDescription struct over list<KeyValue>? I'm afraid that a binary typed field may incur misuse from the users.
There was a problem hiding this comment.
What would the list<KeyValue> contain and where would it reside? I'm not following you.
There was a problem hiding this comment.
struct ExtensionTypeDescription {
1: optional list<KeyValue> metadata
}
And specify the required keys for each extension type, pretty much like what Arrow does.
There was a problem hiding this comment.
This does not make sense, does it? The keys will always be the same, so why not reify them in the Thrift spec as the PR currently does?
There was a problem hiding this comment.
Or are you thinking about extension-specific parameter keys as in https://arrow.apache.org/docs/dev/format/CanonicalExtensions.html ?
There was a problem hiding this comment.
Note we would still need the extension name, so this would be:
struct ExtensionTypeDescription {
1: required string name
2: optional list<KeyValue> parameters
}There was a problem hiding this comment.
Or are you thinking about extension-specific parameter keys as in https://arrow.apache.org/docs/dev/format/CanonicalExtensions.html ?
Yes, I mean something like this.
Rationale for this change
What changes are included in this PR?
Do these changes have PoC implementations?