diff --git a/content/editions/_index.md b/content/editions/_index.md new file mode 100644 index 00000000..7bb0d8b0 --- /dev/null +++ b/content/editions/_index.md @@ -0,0 +1,8 @@ ++++ +title = "Protobuf Editions" +weight = 41 +description = "Topics related to the Protobuf Editions functionality." +type = "docs" ++++ + +* [Protocol Buffers Overview](/editions/overview) diff --git a/content/editions/overview.md b/content/editions/overview.md new file mode 100644 index 00000000..02cea55d --- /dev/null +++ b/content/editions/overview.md @@ -0,0 +1,284 @@ ++++ +title = "Protobuf Editions Overview" +weight = 42 +description = "An overview of the Protobuf Editions functionality." +type = "docs" ++++ + +Protobuf Editions replace the proto2 and proto3 designations that we have used +for Protocol Buffers. Instead of adding `syntax = "proto2"` or `syntax = +"proto3"` at the top of proto definition files, you use an edition number, such +as `edition = "2024"`, to specify the default behaviors your file will have. +Editions enable the language to evolve incrementally over time. + +Instead of the hardcoded behaviors that older versions have had, editions +represent a collection of features with a default value (behavior) per feature. +Features are options on a file, message, field, enum, and so on, that specify +the behavior of protoc, the code generators, and protobuf runtimes. You can +explicitly override a behavior at those different levels (file, message, field, +...) when your needs don't match the default behavior for the edition you've +selected. You can also override your overrides. The +[section later in this topic on inheritance](#inheritance) goes into more detail +on that. + +## Lifecycles of a Feature {#lifecycles} + +Editions provide the fundamental increments for the lifecycle of a feature. +Features have an expected lifecycle: introducing +it, changing its default behavior, deprecating it, and then removing it. For +example: + +1. Edition 2031 creates `feature.amazing_new_feature` with a default value of + `false`. This value maintains the same behavior as all earlier editions. + That is, it defaults to no impact. + +2. Developers update their .proto files to `edition = "2031"`. + +3. A later edition, such as edition 2033, switches the default of + `feature.amazing_new_feature` from `false` to `true`. This is the desired + behavior for all protos, and the reason that the protobuf team created the + feature. + + Using the Prototiller tool to migrate earlier versions of proto files to + edition 2033 adds explicit `feature.amazing_new_feature = false` entries as + needed to continue to retain the previous behavior. Developers remove these + newly-added settings when they want the new behavior to apply to their + .proto files. + + + +5. At some point, `feature.amazing_new_feature` is marked deprecated in an edition and removed in a later one. {value=5} + + When a feature is removed, the code generators for that behavior and the + runtime libraries that support it may also be removed. The timelines will be + generous, though. Following the example in the earlier steps of the + lifecycle, the deprecation might happen in edition 2034 but not be removed + until edition 2036, roughly two years later. Removing a feature will always + initiate a major version bump. + + + +Because of this lifecycle, any `.proto` file that does not use deprecated +features has a no-op upgrade from one edition to the next. +You will have the full +window of the Google migration plus the deprecation window to upgrade your code. + +The preceding lifecycle example used boolean values for the features, but +features may also use enums. For example, `features.field_presence` has values +`LEGACY_REQUIRED`, `EXPLICIT`, and `IMPLICIT.` + +## Migrating to Protobuf Editions {#migrating} + +Editions won't break existing binaries and don't change a message's binary, +text, or JSON serialization format. The first edition is as minimally disruptive +as possible. The first edition establishes the baseline and combines proto2 and +proto3 definitions into a new single definition format. + +When the subsequent editions are released, default behaviors for features may +change. You can have Prototiller do a no-op transformation of your .proto file +or you can choose to accept some or all of the new behaviors. Editions are +planned to be released roughly once a year. + +### Proto2 to Editions {#proto2-migration} + +This section shows a proto2 file, and what it might look like after running the +Prototiller tool to change the definition files to use Protobuf Editions syntax. + +
+ +#### Proto2 syntax {.new-tab} + +```proto +// proto2 file +syntax = "proto2"; + +message Player { + // in proto2, optional fields have explicit presence + optional string name = 1; + // proto2 still supports the problematic "required" field rule + required int32 id = 2; + // in proto2 this is not packed by default + repeated int32 scores = 3; + + enum Handed { + HANDED_UNSPECIFIED = 0, + HANDED_LEFT = 1, + HANDED_RIGHT = 2, + HANDED_AMBIDEXTROUS = 3, + } + + // in proto2 enums are closed + optional Handed handed = 4; +} +``` + +#### Editions syntax {.new-tab} + +```proto +// Edition version of proto2 file +edition = "2023"; + +message Player { + // fields have explicit presence, so no explicit setting needed + string name = 1; + // to match the proto2 behavior, LEGACY_REQUIRED is set at the field level + int32 id = 2 [features.field_presence = LEGACY_REQUIRED]; + // to match the proto2 behavior, EXPANDED is set at the field level + repeated int32 scores = 3 [features.repeated_field_encoding = EXPANDED]; + + enum Handed { + // this overrides the default edition 2023 behavior, which is OPEN + option features.enum_type = CLOSED; + HANDED_UNSPECIFIED = 0, + HANDED_LEFT = 1, + HANDED_RIGHT = 2, + HANDED_AMBIDEXTROUS = 3, + } + + Handed handed = 4; +} +``` + +
+ +### Proto3 to Editions {#proto3-migration} + +This section shows a proto3 file, and what it might look like after running the +Prototiller tool to change the definition files to use Protobuf Editions syntax. + +
+ +#### Proto3 syntax {.new-tab} + +```proto +// proto3 file +syntax = "proto3"; + +message Player { + // in proto3, optional fields have explicit presence + optional string name = 1; + // in proto3 no specified field rule defaults to implicit presence + int32 id = 2; + // in proto3 this is packed by default + repeated int32 scores = 3; + + enum Handed { + HANDED_UNSPECIFIED = 0, + HANDED_LEFT = 1, + HANDED_RIGHT = 2, + HANDED_AMBIDEXTROUS = 3, + } + + // in proto3 enums are open + optional Handed handed = 4; +} +``` + +#### Editions syntax {.new-tab} + +```proto +// Editions version of proto3 file +edition = "2023"; + +message Player { + // fields have explicit presence, so no explicit setting needed + string name = 1; + // to match the proto3 behavior, IMPLICIT is set at the field level + int32 id = 2 [features.field_presence = IMPLICIT]; + // PACKED is the default state, and is provided just for illustration + repeated int32 scores = 3 [features.repeated_field_encoding = PACKED]; + + enum Handed { + HANDED_UNSPECIFIED = 0, + HANDED_LEFT = 1, + HANDED_RIGHT = 2, + HANDED_AMBIDEXTROUS = 3, + } + + Handed handed = 4 [features.field_presence = IMPLICIT]; +} +``` + +
+ +### Inheritance {#inheritance} + +Editions syntax supports inheritance, with a per-feature list of allowed +targets. For example, in the first edition, features can be specified at only +the file level or the lowest level of granularity. Inheritance enables you to +set the default behavior for a feature across an entire file, and then override +that behavior at the message, field, enum, enum value, oneof, service, or +method. Settings made at a higher level (file, message) apply when no setting is +made within the same scope (field, enum value). Any features not explicitly set +conform to the behavior defined in the edition version used for the .proto file. + +The following code sample shows some features being set at the file, message, +and enum level. The settings are in the highlighted lines: + +```proto {highlight="lines:3,7,16"} +edition = "2023"; + +option features.enum_type = CLOSED; + +message Person { + string name = 1; + int32 id = 2 [features.presence = IMPLICIT]; + + enum Pay_Type + PAY_TYPE_UNSPECIFIED = 1, + PAY_TYPE_SALARY = 2, + PAY_TYPE_HOURLY = 3 + } + + enum Employment { + option features.enum_type = OPEN; + EMPLOYMENT_UNSPECIFIED = 0, + EMPLOYMENT_FULLTIME = 1, + EMPLOYMENT_PARTTIME = 2, + } + Employment employment = 4; +} +``` + +In the preceding example, the presence feature is set to `IMPLICIT`; it would +default to `EXPLICIT` if it wasn't set. The `Pay_Type` `enum` will be `CLOSED`, +as it inherits the file-level setting. The `Employment` `enum`, though, will be +`OPEN`, as it is set within the enum. + +### Prototiller {#prototiller} + +We provide both a migration guide and migration tooling that ease the migration +to editions. The tool, called Prototiller, will enable you to: + +* convert proto2 and proto3 definition files to the new editions syntax, at + scale +* migrate files from one edition to another +* manipulate proto files in other ways + +### Backward Compatibility {#compatibility} + +We are building Protobuf Editions to be as minimally disruptive as possible. For +example, you can import proto2 and proto3 definitions into editions-based +definition files, and vice versa: + +```proto +// file myproject/foo.proto +syntax = "proto2"; + +enum Employment { + EMPLOYMENT_UNSPECIFIED = 0, + EMPLOYMENT_FULLTIME = 1, + EMPLOYMENT_PARTTIME = 2, +} +``` + +```proto +// file myproject/edition.proto +edition = "2023"; + +import "myproject/foo.proto"; +``` + +While the generated code changes when you move from proto2 or proto3 to +editions, the wire format does not. You'll still be able to access proto2 and +proto3 data files or file streams using your editions-syntax proto definitions. diff --git a/content/news/2023-06-29.md b/content/news/2023-06-29.md new file mode 100644 index 00000000..8dca443f --- /dev/null +++ b/content/news/2023-06-29.md @@ -0,0 +1,162 @@ ++++ +title = "Changes Announced on June 29, 2023" +weight = 21 +linkTitle = "June 29, 2023" +toc_hide = "true" +description = "Changes announced for Protocol Buffers on June 29, 2023." +type = "docs" ++++ + +**TL;DR:** We are planning to release Protobuf Editions to the open source project in the second half of 2023. While there is no requirement to move from proto2/proto3 syntax to Editions syntax at initial release, we encourage you to plan a move in your software project's future timeline. + +## Protobuf Editions {#editions} + +Protobuf Editions replace the proto2 and proto3 designations that we have used +for Protocol Buffers. Instead of adding `syntax = "proto2"` or `syntax = +"proto3"` at the top of proto definition files, you use an edition number, such +as `edition = "2024"`, to specify the default behaviors your file will have. +Editions enable the language to evolve incrementally over time. + +Instead of the hardcoded behaviors in older versions, editions will represent a +collection of “features” with a default value (behavior) per feature, which you +can override. Features are options on a file, message, field, enum, and so on +that specify the behavior of protoc, the code generators, and protobuf runtimes. +You can explicitly override the desired behavior at those different levels +(file, message, field, ...) when your needs don't match the default behavior for +the edition you've selected. + +Editions won't break existing binaries, and the first edition will be minimally +disruptive; it will establish the baseline and will combine proto2 and proto3 +definitions into a new single definition format. It won't require any changes to +your code. We will be providing a tool, called Prototiller, to migrate .proto +files. The following examples show a proto2 definition file and a proto3 file, +and what each might look like after using Prototiller to convert them to +Protobuf Editions format: + +
+ +#### Proto2 syntax {.new-tab} + +```proto +// proto2 file +syntax = "proto2"; + +message Player { + // in proto2, optional fields have explicit presence + optional string name = 1; + // proto2 still supports the problematic "required" field rule + required int32 id = 2; + // in proto2 this is not packed by default + repeated int32 scores = 3; + + enum Handed { + HANDED_UNSPECIFIED = 0, + HANDED_LEFT = 1, + HANDED_RIGHT = 2, + HANDED_AMBIDEXTROUS = 3, + } + + // in proto2 enums are closed + optional Handed handed = 4; +} +``` + +#### Editions syntax {.new-tab} + +```proto +// Editions version of proto2 file +edition = "2023"; + +message Player { + string name = 1; + int32 id = 2 [features.field_presence = LEGACY_REQUIRED]; + repeated int32 scores = 3 [features.repeated_field_encoding = EXPANDED]; + + enum Handed { + // this overrides the default Edition 2023 behavior, which is OPEN + option features.enum = CLOSED; + HANDED_UNSPECIFIED = 0, + HANDED_LEFT = 1, + HANDED_RIGHT = 2, + HANDED_AMBIDEXTROUS = 3, + } + + Handed handed = 4; +} +``` + +
+ +And this is what a similar proto3 definition file might look like: + +
+ +#### Proto3 syntax {.new-tab} + +```proto +// proto3 file +syntax = "proto3"; + +message Player { + // in proto3, optional fields have explicit presence + optional string name = 1; + // in proto3 no specified field rule defaults to implicit presence + int32 id = 2; + // in proto3 this is packed by default + repeated int32 scores = 3; + + enum Handed { + HANDED_UNSPECIFIED = 0, + HANDED_LEFT = 1, + HANDED_RIGHT = 2, + HANDED_AMBIDEXTROUS = 3, + } + + // in proto3 enums are open + optional Handed handed = 4; +} +``` + +#### Editions syntax {.new-tab} + +```proto +// Editions version of proto3 file +edition = "2023"; + +message Player { + string name = 1; + int32 id = 2 [features.field_presence = IMPLICIT]; + repeated int32 scores = 3; + + enum Handed { + HANDED_UNSPECIFIED = 0, + HANDED_LEFT = 1, + HANDED_RIGHT = 2, + HANDED_AMBIDEXTROUS = 3, + } + + Handed handed = 4 [features.field_presence = IMPLICIT]; +} +``` + +
+ +While the examples provided in this topic show a direct translation of proto2 +and proto3 to the equivalent representation using Protobuf Editions, you will be +able to mix and match the settings to your project's needs. + +Features have a lifecycle that is governed by the releases of editions. For +example, `features.awesome_new_feature` might be added in Edition 2031, with the +new behavior applying to all definitions that don't explicitly override the new +behavior. In Edition 2033, the new feature is deprecated. Overrides still work, +but developers are alerted to the fact that they need to adapt to the new +behavior soon. In Edition 2036, the feature is removed and the new behavior +applies to all protos; there is no way to override the new behavior at this +point. + +![Editions lifecycle](/images/protobuf-editions-lifecycle-short.png "Editions lifecycle") +\ +**Figure 1: Editions lifecycle flowchart** + +Editions are planned to be released roughly once a year. For more information on +Protobuf Editions, see the overview at https://protobuf.dev/editions/overview. diff --git a/content/news/_index.md b/content/news/_index.md index cb09451a..ab7b5280 100644 --- a/content/news/_index.md +++ b/content/news/_index.md @@ -8,6 +8,8 @@ type = "docs" News topics provide information about past events and changes with Protocol Buffers, and plans for upcoming changes. +* [June 29, 2023](/news/2023-06-29) - Protobuf Editions + announcement * [April 28, 2023](/news/2023-04-28) - Null no longer allowed in `json_name` field options * [April 20, 2023](/news/2023-04-20) - Update to Ruby diff --git a/content/programming-guides/encoding.md b/content/programming-guides/encoding.md index e7fa3bcd..265df609 100644 --- a/content/programming-guides/encoding.md +++ b/content/programming-guides/encoding.md @@ -198,14 +198,19 @@ Signed Original | Encoded As 0x7fffffff | 0xfffffffe -0x80000000 | 0xffffffff -Using some bit tricks, it's cheap to convert `n` into its ZigZag representation: +In other words, each value `n` is encoded using ``` -((n + n) ^ -(n < 0)) - (n < 0) +(n << 1) ^ (n >> 31) ``` -Here, we assume that the boolean `n < 0` is converted into an integer 1 if true -or an integer 0 if false. +for `sint32`s, or + +``` +(n << 1) ^ (n >> 63) +``` + +for the 64-bit version. When the `sint32` or `sint64` is parsed, its value is decoded back to the original, signed version. diff --git a/content/support/version-support.md b/content/support/version-support.md index 23a8326d..dedfe27b 100644 --- a/content/support/version-support.md +++ b/content/support/version-support.md @@ -38,7 +38,7 @@ defined by our The most recent release is always supported. Support for earlier minor versions ends when a new minor version under the same major version is released. Support -for earlier major verions ends four quarters beyond the quarter that the +for earlier major versions ends four quarters beyond the quarter that the breaking release is introduced. For example, when Python 4.21.0 was released in May of 2022, that set the end of public support of Python 3.20.1 at the [end of 2023 Q2](#python). @@ -180,7 +180,7 @@ Protobuf is committed to following the tooling, platform, and library support policy described in [Foundational C++ Support Policy](https://opensource.google/documentation/policies/cplusplus-support). For specific versions supported, see -[Foundational C++ Support](https://github.com/google/oss-policies-info/blob/main/foundational-cxx-support-matrix.md). +[Foundational C++ Support Matrix](https://github.com/google/oss-policies-info/blob/main/foundational-cxx-support-matrix.md). ## C# {#csharp} @@ -322,21 +322,6 @@ The Java 3.23.x runtime was first released in 2023 Q2. 24Q2 24Q3 - - 19.x - 3.19.x - - - - - - - - - - - - 20.x 3.20.x @@ -823,6 +808,12 @@ The Python 3.20.x runtime was first released in 2022 Q1 and has support until +### Python Platform and Library Support {#python-support} + +Protobuf is committed to following the platform and library support policy +described in +[Python Support Policy](https://cloud.google.com/python/docs/supported-python-versions). + ## Ruby {#ruby} The Ruby 3.23.x runtime was first released in 2023 Q2. @@ -950,3 +941,11 @@ The Ruby 3.23.x runtime was first released in 2023 Q2. + +### Ruby Platform and Library Support {#ruby-support} + +Protobuf is committed to following the platform and library support policy +described in +[Ruby Support Policy](https://cloud.google.com/ruby/getting-started/supported-ruby-versions). +For specific versions supported, see +[Foundational Ruby Support Matrix](https://github.com/google/oss-policies-info/blob/main/foundational-ruby-support-matrix.md).