PARQUET-2480: Clarify what "page index" means in Parquet.thrift

[alamb]

https://issues.apache.org/jira/browse/PARQUET-2480

See the proposed update as rendered markdown: https://github.com/alamb/parquet-format/blob/alamb/page-index/PageIndex.md

I have always found it very confusing that people refer to the term parquet "page index", for example this message

However, the term "page index" is not used in the the parquet.thrift file itself, but only appears as the name of the file that describes the ColumnIndex and OffsetIndex, PageIndex.md

This means I can't search for "page index" in the spec and find out what people are talking about

Proposed Clarifications

1. Update the introductory paragraph of PageIndex.md to clarify use the term "page index" and explain that it is encoded as ColumnIndex and OffsetIndex
2. Update the description of ColumnIndex and OffsetIndex to include the term "page index" and clarify what those structures are used for.

Jira

• My PR addresses the following Parquet Jira issues and references them in the PR title. For example, "PARQUET-1234: My Parquet PR"
  • https://issues.apache.org/jira/browse/PARQUET-2480
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Commits

• My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

This PR has no spec changes, only clarifications

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain Javadoc that explain what it does

[mapleFU]

I also think that currently PageIndex means "offset index and column index". They're all page-level index

[alamb]

I updated this to avoid potential confusion with ColumnIndex which is a different structure

[alamb]

Thank you for the comments @mapleFU @tustvold and @wgtmac - I believe I have implemented your suggestions and I think the PR is much clearer because of it.

[mapleFU]

General LGTM

[mapleFU]

With statistics enabled, only OffsetIndex is presented, with can be used to planning IO and get first_row_ordinal for ColumnChunk?

[alamb]

I am sorry @mapleFU -- I don't quite follow this suggestion

Do you mean that some writers may write only OffsetIndex and that both structures may not be present?

[wgtmac]

Yes, ColumnIndex will not be created when there is no valid min/max statistics. In parquet-cpp, we can also disable creating ColumnIndex but enabling OffsetIndex.

[mapleFU]

Yeah, I guess OffsetIndex might be used separately ( But personally I don't think Column Index might be used like that, lol)

[alamb]

I added

--- a/src/main/thrift/parquet.thrift
+++ b/src/main/thrift/parquet.thrift
@@ -1005,6 +1005,8 @@ struct PageLocation {
  * Optional offsets for each data page in a ColumnChunk.
  *
  * Forms part of the page index, along with ColumnIndex.
+ *
+ * OffsetIndex may be present even if ColumnIndex is not.
  */
 struct OffsetIndex {
   /**
@@ -1026,6 +1028,8 @@ struct OffsetIndex {
  *
  * Forms part the page index, along with OffsetIndex.
  *
+ * If this structure is present, OffsetIndex must also be present.
+ *
  * For each field in this structure, <field>[i] refers to the page at
  * OffsetIndex.page_locations[i]
  */

in 8d59c7d to try and clarify this

[wgtmac]

LGTM +1

[mapleFU]

LGTM, thanks!

[wgtmac]

@gszadovszky @pitrou @emkornfield @julienledem Would you like to take a look?