Skip to content

Commit

Permalink
PARQUET-758: Add Float16/Half-float logical type (#184)
Browse files Browse the repository at this point in the history
Type involves a trade-off of reduced precision,
in exchange for more efficient storage.
  • Loading branch information
anjakefala committed Oct 27, 2023
1 parent 31f92c7 commit 46cc3a0
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 0 deletions.
10 changes: 10 additions & 0 deletions LogicalTypes.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,16 @@ comparison.
To support compatibility with older readers, implementations of parquet-format should
write `DecimalType` precision and scale into the corresponding SchemaElement field in metadata.

### FLOAT16

The `FLOAT16` annotation represents half-precision floating-point numbers in the 2-byte IEEE little-endian format.

Used in contexts where precision is traded off for smaller footprint and potentially better performance.

The primitive type is a 2-byte fixed length binary.

The sort order for `FLOAT16` is signed (with special handling of NANs and signed zeros); it uses the same [logic](https://github.com/apache/parquet-format#sort-order) as `FLOAT` and `DOUBLE`.

## Temporal Types

### DATE
Expand Down
2 changes: 2 additions & 0 deletions src/main/thrift/parquet.thrift
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,7 @@ struct MapType {} // see LogicalTypes.md
struct ListType {} // see LogicalTypes.md
struct EnumType {} // allowed for BINARY, must be encoded with UTF-8
struct DateType {} // allowed for INT32
struct Float16Type {} // allowed for FIXED[2], must encoded raw FLOAT16 bytes

/**
* Logical type to annotate a column that is always null.
Expand Down Expand Up @@ -355,6 +356,7 @@ union LogicalType {
12: JsonType JSON // use ConvertedType JSON
13: BsonType BSON // use ConvertedType BSON
14: UUIDType UUID // no compatible ConvertedType
15: Float16Type FLOAT16 // no compatible ConvertedType
}

/**
Expand Down

0 comments on commit 46cc3a0

Please sign in to comment.