Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support reading maps with map vs key_value #96

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Support reading maps with map vs key_value #96

wants to merge 2 commits into from

Commits on Nov 30, 2022

  1. Failing test.

    @parsnips
    parsnips committed Nov 30, 2022
    Configuration menu
    Copy the full SHA
    3cd6c74 View commit details
    Browse the repository at this point in the history

  2. support reading maps with map instead of key_value 

    Older writer libraries will write maps in the [old
format](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules-1).

Notably, the Kinesis Firehose
[ParquetSerDe](https://docs.aws.amazon.com/firehose/latest/APIReference/API_ParquetSerDe.html)

This add support to read those types.

You can find a sample file at
https://gist.github.com/parsnips/928c14d850331dd21c8d917227f77c5c to
run in the test
    @parsnips
    parsnips committed Nov 30, 2022
    Configuration menu
    Copy the full SHA
    8f0267e View commit details
    Browse the repository at this point in the history