Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revert "Apply auto-flattening to subobjects: auto (#112092)" #113692

Merged
merged 4 commits into from
Sep 30, 2024
Merged

Revert "Apply auto-flattening to subobjects: auto (#112092)" #113692

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8d51283
Revert "Apply auto-flattening to `subobjects: auto` (#112092)"
kkrik-es Sep 27, 2024
f4322f8
Merge branch 'main' into rollback/subobjects-auto-flatten
kkrik-es Sep 29, 2024
72bbd24
Merge branch 'main' into rollback/subobjects-auto-flatten
kkrik-es Sep 30, 2024
a0e66e6
fix DataGenerationHelper
kkrik-es Sep 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions docs/changelog/112092.yaml
View file
Open in desktop

This file was deleted.

224 changes: 16 additions & 208 deletions docs/reference/mapping/params/subobjects.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ where for instance a field `metrics.time` holds a value too, which is common whe
A document holding a value for both `metrics.time.max` and `metrics.time` gets rejected given that `time`
would need to be a leaf field to hold a value as well as an object to hold the `max` sub-field.

The `subobjects: false` setting, which can be applied only to the top-level mapping definition and
The `subobjects` setting, which can be applied only to the top-level mapping definition and
to <<object,`object`>> fields, disables the ability for an object to hold further subobjects and makes it possible
to store documents where field names contain dots and share common prefixes. From the example above, if the object
container `metrics` has `subobjects` set to `false`, it can hold values for both `time` and `time.max` directly
Expand Down Expand Up @@ -109,138 +109,26 @@ PUT my-index-000001/_doc/metric_1
<1> The entire mapping is configured to not support objects.
<2> The document does not support objects

Setting `subobjects: false` disallows the definition of <<object,`object`>> and <<nested,`nested`>> sub-fields, which
can be too restrictive in cases where it's desirable to have <<nested,`nested`>> objects or sub-objects with specific
behavior (e.g. with `enabled:false`). In this case, it's possible to set `subobjects: auto`, which
<<auto-flattening, auto-flattens>> whenever possible and falls back to creating an object mapper otherwise (instead of
rejecting the mapping as `subobjects: false` does). For instance:

[source,console]
--------------------------------------------------
PUT my-index-000002
{
"mappings": {
"properties": {
"metrics": {
"type": "object",
"subobjects": "auto", <1>
"properties": {
"inner": {
"type": "object",
"enabled": false
},
"nested": {
"type": "nested"
}
}
}
}
}
}

PUT my-index-000002/_doc/metric_1
{
"metrics.time" : 100, <2>
"metrics.time.min" : 10,
"metrics.time.max" : 900
}

PUT my-index-000002/_doc/metric_2
{
"metrics" : { <3>
"time" : 100,
"time.min" : 10,
"time.max" : 900,
"inner": {
"foo": "bar",
"path.to.some.field": "baz"
},
"nested": [
{ "id": 10 },
{ "id": 1 }
]
}
}

GET my-index-000002/_mapping
--------------------------------------------------

[source,console-result]
--------------------------------------------------
{
"my-index-000002" : {
"mappings" : {
"properties" : {
"metrics" : {
"subobjects" : auto,
"properties" : {
"inner": { <4>
"type": "object",
"enabled": false
},
"nested": {
"type": "nested",
"properties" : {
"id" : {
"type" : "long"
}
}
},
"time" : {
"type" : "long"
},
"time.min" : {
"type" : "long"
},
"time.max" : {
"type" : "long"
}
}
}
}
}
}
}
--------------------------------------------------

<1> The `metrics` field can only hold statically defined objects, namely `inner` and `nested`.
<2> Sample document holding flat paths
<3> Sample document holding an object (configured with sub-objects) and its leaf sub-fields
<4> The resulting mapping where dots in field names (`time.min`, `time_max`), as well as the
statically-defined sub-objects `inner` and `nested`, were preserved

The `subobjects` setting for existing fields and the top-level mapping definition cannot be updated.

[[auto-flattening]]
==== Auto-flattening object mappings

It is generally recommended to define the properties of an object that is configured with `subobjects: false` or
`subobjects: auto` with dotted field names (as shown in the first example). However, it is also possible to define
these properties as sub-objects in the mappings. In that case, the mapping will be automatically flattened before
it is stored. This makes it easier to re-use existing mappings without having to re-write them.

Note that auto-flattening does not apply if any of the following <<mapping-params, mapping parameters>> are set
on object mappings that are defined under an object configured with `subobjects: false` or `subobjects: auto`:
It is generally recommended to define the properties of an object that is configured with `subobjects: false` with dotted field names
(as shown in the first example).
However, it is also possible to define these properties as sub-objects in the mappings.
In that case, the mapping will be automatically flattened before it is stored.
This makes it easier to re-use existing mappings without having to re-write them.

* The <<enabled, `enabled`>> mapping parameter is `false`.
* The <<dynamic, `dynamic`>> mapping parameter contradicts the implicit or explicit value of the parent.
For example, when `dynamic` is set to `false` in the root of the mapping, object mappers that set `dynamic` to `true`
can't be auto-flattened.
* The <<subobjects, `subobjects`>> mapping parameter is set to `auto` or `true` explicitly.
Note that auto-flattening will not work when certain <<mapping-params, mapping parameters>> are set
on object mappings that are defined under an object configured with `subobjects: false`:

If such a sub-object is detected, the behavior depends on the `subobjects` value:

* `subobjects: false` is not compatible, so a mapping error is returned during mapping construction.
* `subobjects: auto` reverts to adding the object to the mapping, bypassing auto-flattening for it. Still, any
intermediate objects will be auto-flattened if applicable (i.e. the object name gets directly attached under the parent
object with `subobjects: auto`). Auto-flattening can be applied within sub-objects, if they are configured with
`subobjects: auto` too.

Auto-flattening example with `subobjects: false`:
* The <<enabled, `enabled`>> mapping parameter must not be `false`.
* The <<dynamic, `dynamic`>> mapping parameter must not contradict the implicit or explicit value of the parent. For example, when `dynamic` is set to `false` in the root of the mapping, object mappers that set `dynamic` to `true` can't be auto-flattened.
* The <<subobjects, `subobjects`>> mapping parameter must not be set to `true` explicitly.

[source,console]
--------------------------------------------------
PUT my-index-000003
PUT my-index-000002
{
"mappings": {
"properties": {
Expand All @@ -259,13 +147,13 @@ PUT my-index-000003
}
}
}
GET my-index-000003/_mapping
GET my-index-000002/_mapping
--------------------------------------------------

[source,console-result]
--------------------------------------------------
{
"my-index-000003" : {
"my-index-000002" : {
"mappings" : {
"properties" : {
"metrics" : {
Expand All @@ -287,85 +175,5 @@ GET my-index-000003/_mapping

<1> The metrics object can contain further object mappings that will be auto-flattened.
Object mappings at this level must not set certain mapping parameters as explained above.
<2> This field will be auto-flattened to `time.min` before the mapping is stored.
<3> The auto-flattened `time.min` field can be inspected by looking at the index mapping.

Auto-flattening example with `subobjects: auto`:

[source,console]
--------------------------------------------------
PUT my-index-000004
{
"mappings": {
"properties": {
"metrics": {
"subobjects": "auto",
"properties": {
"time": {
"type": "object", <1>
"properties": {
"min": { "type": "long" } <2>
}
},
"to": {
"type": "object",
"properties": {
"inner_metrics": { <3>
"type": "object",
"subobjects": "auto",
"properties": {
"time": {
"type": "object",
"properties": {
"max": { "type": "long" } <4>
}
}
}
}
}
}
}
}
}
}
}
GET my-index-000004/_mapping
--------------------------------------------------

[source,console-result]
--------------------------------------------------
{
"my-index-000004" : {
"mappings" : {
"properties" : {
"metrics" : {
"subobjects" : "auto",
"properties" : {
"time.min" : { <5>
"type" : "long"
},
"to.inner_metrics" : { <6>
"subobjects" : "auto",
"properties" : {
"time.max" : { <7>
"type" : "long"
}
}
}
}
}
}
}
}
}
--------------------------------------------------

<1> The metrics object can contain further object mappings that may be auto-flattened, depending on their mapping
parameters as explained above.
<2> This field will be auto-flattened to `time.min` before the mapping is stored.
<3> This object has param `subobjects: auto` so it can't be auto-flattened. Its parent does qualify for auto-flattening,
so it becomes `to.inner_metrics` before the mapping is stored.
<4> This field will be auto-flattened to `time.max` before the mapping is stored.
<5> The auto-flattened `time.min` field can be inspected by looking at the index mapping.
<6> The inner object `to.inner_metrics` can be inspected by looking at the index mapping.
<7> The auto-flattened `time.max` field can be inspected by looking at the index mapping.
<2> This field will be auto-flattened to `"time.min"` before the mapping is stored.
<3> The auto-flattened `"time.min"` field can be inspected by looking at the index mapping.
7 changes: 6 additions & 1 deletion ...s/src/javaRestTest/java/org/elasticsearch/datastreams/logsdb/qa/DataGenerationHelper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,12 @@ public DataGenerationHelper() {
}

public DataGenerationHelper(Consumer<DataGeneratorSpecification.Builder> builderConfigurator) {
this.subobjects = ESTestCase.randomFrom(ObjectMapper.Subobjects.values());
// TODO enable subobjects: auto
// It is disabled because it currently does not have auto flattening and that results in asserts being triggered when using copy_to.
this.subobjects = ESTestCase.randomValueOtherThan(
ObjectMapper.Subobjects.AUTO,
() -> ESTestCase.randomFrom(ObjectMapper.Subobjects.values())
);
this.keepArraySource = ESTestCase.randomBoolean();

var specificationBuilder = DataGeneratorSpecification.builder().withFullyDynamicMapping(ESTestCase.randomBoolean());
Expand Down
8 changes: 4 additions & 4 deletions ...i-spec/src/yamlRestTest/resources/rest-api-spec/test/index/92_metrics_auto_subobjects.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"Metrics object indexing":
- requires:
test_runner_features: [ "allowed_warnings", "allowed_warnings_regex" ]
cluster_features: ["mapper.subobjects_auto_fixes"]
cluster_features: ["mapper.subobjects_auto"]
reason: requires supporting subobjects auto setting

- do:
Expand Down Expand Up @@ -69,7 +69,7 @@
"Root with metrics":
- requires:
test_runner_features: [ "allowed_warnings", "allowed_warnings_regex" ]
cluster_features: ["mapper.subobjects_auto_fixes"]
cluster_features: ["mapper.subobjects_auto"]
reason: requires supporting subobjects auto setting

- do:
Expand Down Expand Up @@ -131,7 +131,7 @@
"Metrics object indexing with synthetic source":
- requires:
test_runner_features: [ "allowed_warnings", "allowed_warnings_regex" ]
cluster_features: ["mapper.subobjects_auto_fixes"]
cluster_features: ["mapper.subobjects_auto"]
reason: added in 8.4.0

- do:
Expand Down Expand Up @@ -201,7 +201,7 @@
"Root without subobjects with synthetic source":
- requires:
test_runner_features: [ "allowed_warnings", "allowed_warnings_regex" ]
cluster_features: ["mapper.subobjects_auto_fixes"]
cluster_features: ["mapper.subobjects_auto"]
reason: added in 8.4.0

- do:
Expand Down
Loading