Skip to content

Commit

Permalink
Revert "Apply auto-flattening to subobjects: auto (#112092)" (#113692)
Browse files Browse the repository at this point in the history 
* Revert "Apply auto-flattening to `subobjects: auto` (#112092)"

This reverts commit fffe884

* fix DataGenerationHelper

(cherry picked from commit c9f378d)

# Conflicts:
#	server/src/main/java/org/elasticsearch/index/mapper/DocumentParserContext.java
  • Loading branch information
@kkrik-es
kkrik-es committed Sep 30, 2024
1 parent 2fc2d05 commit fdf29f0
Show file tree
Hide file tree
Showing 18 changed files with 219 additions and 1,048 deletions.
5 changes: 0 additions & 5 deletions docs/changelog/112092.yaml
View file

This file was deleted.

224 changes: 16 additions & 208 deletions docs/reference/mapping/params/subobjects.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ where for instance a field `metrics.time` holds a value too, which is common whe
A document holding a value for both `metrics.time.max` and `metrics.time` gets rejected given that `time`
would need to be a leaf field to hold a value as well as an object to hold the `max` sub-field.

The `subobjects: false` setting, which can be applied only to the top-level mapping definition and
The `subobjects` setting, which can be applied only to the top-level mapping definition and
to <<object,`object`>> fields, disables the ability for an object to hold further subobjects and makes it possible
to store documents where field names contain dots and share common prefixes. From the example above, if the object
container `metrics` has `subobjects` set to `false`, it can hold values for both `time` and `time.max` directly
Expand Down Expand Up @@ -109,138 +109,26 @@ PUT my-index-000001/_doc/metric_1
<1> The entire mapping is configured to not support objects.
<2> The document does not support objects

Setting `subobjects: false` disallows the definition of <<object,`object`>> and <<nested,`nested`>> sub-fields, which
can be too restrictive in cases where it's desirable to have <<nested,`nested`>> objects or sub-objects with specific
behavior (e.g. with `enabled:false`). In this case, it's possible to set `subobjects: auto`, which
<<auto-flattening, auto-flattens>> whenever possible and falls back to creating an object mapper otherwise (instead of
rejecting the mapping as `subobjects: false` does). For instance:

[source,console]
--------------------------------------------------
PUT my-index-000002
{
"mappings": {
"properties": {
"metrics": {
"type": "object",
"subobjects": "auto", <1>
"properties": {
"inner": {
"type": "object",
"enabled": false
},
"nested": {
"type": "nested"
}
}
}
}
}
}
PUT my-index-000002/_doc/metric_1
{
"metrics.time" : 100, <2>
"metrics.time.min" : 10,
"metrics.time.max" : 900
}
PUT my-index-000002/_doc/metric_2
{
"metrics" : { <3>
"time" : 100,
"time.min" : 10,
"time.max" : 900,
"inner": {
"foo": "bar",
"path.to.some.field": "baz"
},
"nested": [
{ "id": 10 },
{ "id": 1 }
]
}
}
GET my-index-000002/_mapping
--------------------------------------------------

[source,console-result]
--------------------------------------------------
{
"my-index-000002" : {
"mappings" : {
"properties" : {
"metrics" : {
"subobjects" : auto,
"properties" : {
"inner": { <4>
"type": "object",
"enabled": false
},
"nested": {
"type": "nested",
"properties" : {
"id" : {
"type" : "long"
}
}
},
"time" : {
"type" : "long"
},
"time.min" : {
"type" : "long"
},
"time.max" : {
"type" : "long"
}
}
}
}
}
}
}
--------------------------------------------------

<1> The `metrics` field can only hold statically defined objects, namely `inner` and `nested`.
<2> Sample document holding flat paths
<3> Sample document holding an object (configured with sub-objects) and its leaf sub-fields
<4> The resulting mapping where dots in field names (`time.min`, `time_max`), as well as the
statically-defined sub-objects `inner` and `nested`, were preserved

The `subobjects` setting for existing fields and the top-level mapping definition cannot be updated.

[[auto-flattening]]
==== Auto-flattening object mappings

It is generally recommended to define the properties of an object that is configured with `subobjects: false` or
`subobjects: auto` with dotted field names (as shown in the first example). However, it is also possible to define
these properties as sub-objects in the mappings. In that case, the mapping will be automatically flattened before
it is stored. This makes it easier to re-use existing mappings without having to re-write them.

Note that auto-flattening does not apply if any of the following <<mapping-params, mapping parameters>> are set
on object mappings that are defined under an object configured with `subobjects: false` or `subobjects: auto`:
It is generally recommended to define the properties of an object that is configured with `subobjects: false` with dotted field names
(as shown in the first example).
However, it is also possible to define these properties as sub-objects in the mappings.
In that case, the mapping will be automatically flattened before it is stored.
This makes it easier to re-use existing mappings without having to re-write them.

* The <<enabled, `enabled`>> mapping parameter is `false`.
* The <<dynamic, `dynamic`>> mapping parameter contradicts the implicit or explicit value of the parent.
For example, when `dynamic` is set to `false` in the root of the mapping, object mappers that set `dynamic` to `true`
can't be auto-flattened.
* The <<subobjects, `subobjects`>> mapping parameter is set to `auto` or `true` explicitly.
Note that auto-flattening will not work when certain <<mapping-params, mapping parameters>> are set
on object mappings that are defined under an object configured with `subobjects: false`:

If such a sub-object is detected, the behavior depends on the `subobjects` value:

* `subobjects: false` is not compatible, so a mapping error is returned during mapping construction.
* `subobjects: auto` reverts to adding the object to the mapping, bypassing auto-flattening for it. Still, any
intermediate objects will be auto-flattened if applicable (i.e. the object name gets directly attached under the parent
object with `subobjects: auto`). Auto-flattening can be applied within sub-objects, if they are configured with
`subobjects: auto` too.

Auto-flattening example with `subobjects: false`:
* The <<enabled, `enabled`>> mapping parameter must not be `false`.
* The <<dynamic, `dynamic`>> mapping parameter must not contradict the implicit or explicit value of the parent. For example, when `dynamic` is set to `false` in the root of the mapping, object mappers that set `dynamic` to `true` can't be auto-flattened.
* The <<subobjects, `subobjects`>> mapping parameter must not be set to `true` explicitly.

[source,console]
--------------------------------------------------
PUT my-index-000003
PUT my-index-000002
{
"mappings": {
"properties": {
Expand All @@ -259,13 +147,13 @@ PUT my-index-000003
}
}
}
GET my-index-000003/_mapping
GET my-index-000002/_mapping
--------------------------------------------------

[source,console-result]
--------------------------------------------------
{
"my-index-000003" : {
"my-index-000002" : {
"mappings" : {
"properties" : {
"metrics" : {
Expand All @@ -287,85 +175,5 @@ GET my-index-000003/_mapping

<1> The metrics object can contain further object mappings that will be auto-flattened.
Object mappings at this level must not set certain mapping parameters as explained above.
<2> This field will be auto-flattened to `time.min` before the mapping is stored.
<3> The auto-flattened `time.min` field can be inspected by looking at the index mapping.

Auto-flattening example with `subobjects: auto`:

[source,console]
--------------------------------------------------
PUT my-index-000004
{
"mappings": {
"properties": {
"metrics": {
"subobjects": "auto",
"properties": {
"time": {
"type": "object", <1>
"properties": {
"min": { "type": "long" } <2>
}
},
"to": {
"type": "object",
"properties": {
"inner_metrics": { <3>
"type": "object",
"subobjects": "auto",
"properties": {
"time": {
"type": "object",
"properties": {
"max": { "type": "long" } <4>
}
}
}
}
}
}
}
}
}
}
}
GET my-index-000004/_mapping
--------------------------------------------------

[source,console-result]
--------------------------------------------------
{
"my-index-000004" : {
"mappings" : {
"properties" : {
"metrics" : {
"subobjects" : "auto",
"properties" : {
"time.min" : { <5>
"type" : "long"
},
"to.inner_metrics" : { <6>
"subobjects" : "auto",
"properties" : {
"time.max" : { <7>
"type" : "long"
}
}
}
}
}
}
}
}
}
--------------------------------------------------

<1> The metrics object can contain further object mappings that may be auto-flattened, depending on their mapping
parameters as explained above.
<2> This field will be auto-flattened to `time.min` before the mapping is stored.
<3> This object has param `subobjects: auto` so it can't be auto-flattened. Its parent does qualify for auto-flattening,
so it becomes `to.inner_metrics` before the mapping is stored.
<4> This field will be auto-flattened to `time.max` before the mapping is stored.
<5> The auto-flattened `time.min` field can be inspected by looking at the index mapping.
<6> The inner object `to.inner_metrics` can be inspected by looking at the index mapping.
<7> The auto-flattened `time.max` field can be inspected by looking at the index mapping.
<2> This field will be auto-flattened to `"time.min"` before the mapping is stored.
<3> The auto-flattened `"time.min"` field can be inspected by looking at the index mapping.
7 changes: 6 additions & 1 deletion ...s/src/javaRestTest/java/org/elasticsearch/datastreams/logsdb/qa/DataGenerationHelper.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,12 @@ public DataGenerationHelper() {
}

public DataGenerationHelper(Consumer<DataGeneratorSpecification.Builder> builderConfigurator) {
this.subobjects = ESTestCase.randomFrom(ObjectMapper.Subobjects.values());
// TODO enable subobjects: auto
// It is disabled because it currently does not have auto flattening and that results in asserts being triggered when using copy_to.
this.subobjects = ESTestCase.randomValueOtherThan(
ObjectMapper.Subobjects.AUTO,
() -> ESTestCase.randomFrom(ObjectMapper.Subobjects.values())
);
this.keepArraySource = ESTestCase.randomBoolean();

var specificationBuilder = DataGeneratorSpecification.builder().withFullyDynamicMapping(ESTestCase.randomBoolean());
Expand Down
8 changes: 4 additions & 4 deletions ...i-spec/src/yamlRestTest/resources/rest-api-spec/test/index/92_metrics_auto_subobjects.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"Metrics object indexing":
- requires:
test_runner_features: [ "allowed_warnings", "allowed_warnings_regex" ]
cluster_features: ["mapper.subobjects_auto_fixes"]
cluster_features: ["mapper.subobjects_auto"]
reason: requires supporting subobjects auto setting

- do:
Expand Down Expand Up @@ -69,7 +69,7 @@
"Root with metrics":
- requires:
test_runner_features: [ "allowed_warnings", "allowed_warnings_regex" ]
cluster_features: ["mapper.subobjects_auto_fixes"]
cluster_features: ["mapper.subobjects_auto"]
reason: requires supporting subobjects auto setting

- do:
Expand Down Expand Up @@ -131,7 +131,7 @@
"Metrics object indexing with synthetic source":
- requires:
test_runner_features: [ "allowed_warnings", "allowed_warnings_regex" ]
cluster_features: ["mapper.subobjects_auto_fixes"]
cluster_features: ["mapper.subobjects_auto"]
reason: added in 8.4.0

- do:
Expand Down Expand Up @@ -201,7 +201,7 @@
"Root without subobjects with synthetic source":
- requires:
test_runner_features: [ "allowed_warnings", "allowed_warnings_regex" ]
cluster_features: ["mapper.subobjects_auto_fixes"]
cluster_features: ["mapper.subobjects_auto"]
reason: added in 8.4.0

- do:
Expand Down
Loading

0 comments on commit fdf29f0

Please sign in to comment.