Collect and display execution metadata for ES|QL cross cluster searches #112595
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| pr: 112595 | ||
| summary: Collect and display execution metadata for ES|QL cross cluster searches | ||
| area: ES|QL | ||
| type: enhancement | ||
| issues: | ||
| - 112402 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -85,7 +85,7 @@ POST /_security/role/remote1 | |
| "privileges": [ "read","read_cross_cluster" ], <4> | ||
| "clusters" : ["my_remote_cluster"] <5> | ||
| } | ||
| ], | ||
| "remote_cluster": [ <6> | ||
| { | ||
| "privileges": [ | ||
|
|
@@ -100,15 +100,23 @@ POST /_security/role/remote1 | |
| ---- | ||
|
|
||
| <1> The `cross_cluster_search` cluster privilege is required for the _local_ cluster. | ||
| <2> Typically, users will have permissions to read both local and remote indices. However, for cases where the role | ||
| is intended to ONLY search the remote cluster, the `read` permission is still required for the local cluster. | ||
| To provide read access to the local cluster, but disallow reading any indices in the local cluster, the `names` | ||
| field may be an empty string. | ||
| <3> The indices allowed read access to the remote cluster. The configured | ||
| <<security-api-create-cross-cluster-api-key,cross-cluster API key>> must also allow this index to be read. | ||
| <4> The `read_cross_cluster` privilege is always required when using {esql} across clusters with the API key based | ||
| security model. | ||
| <5> The remote clusters to which these privileges apply. | ||
| This remote cluster must be configured with a <<security-api-create-cross-cluster-api-key,cross-cluster API key>> | ||
| and connected to the remote cluster before the remote index can be queried. | ||
| Verify connection using the <<cluster-remote-info, Remote cluster info>> API. | ||
| <6> Required to allow remote enrichment. Without this, the user cannot read from the `.enrich` indices on the | ||
| remote cluster. The `remote_cluster` security privilege was introduced in version *8.15.0*. | ||
|
|
||
| You will then need a user or API key with the permissions you created above. The following example API call creates | ||
| a user with the `remote1` role. | ||
|
|
||
| [source,console] | ||
| ---- | ||
|
|
@@ -119,11 +127,13 @@ POST /_security/user/remote_user | |
| } | ||
| ---- | ||
|
|
||
| Remember that all cross-cluster requests from the local cluster are bound by the cross cluster API key’s privileges, | ||
| which are controlled by the remote cluster's administrator. | ||
|
|
||
| [TIP] | ||
| ==== | ||
| Cross cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to add the new permissions | ||
| required for {esql} with ENRICH. | ||
| ==== | ||
|
|
||
| [discrete] | ||
|
|
@@ -174,6 +184,194 @@ FROM *:my-index-000001 | |
| | LIMIT 10 | ||
| ---- | ||
|
|
||
| [discrete] | ||
| [[ccq-cluster-details]] | ||
| ==== Cross-cluster metadata | ||
|
|
||
| ES|QL {ccs} responses include metadata about the search on each cluster when the response format is JSON. | ||
| Here we show an example using the async search endpoint. {ccs-cap} metadata is also present in the synchronous | ||
| search endpoint. | ||
|
|
||
| [source,console] | ||
| ---- | ||
| POST /_query/async?format=json | ||
| { | ||
| "query": """ | ||
| FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index* | ||
| | STATS COUNT(http.response.status_code) BY user.id | ||
| | LIMIT 2 | ||
| """ | ||
| } | ||
| ---- | ||
| // TEST[setup:my_index] | ||
| // TEST[s/cluster_one:my-index-000001,cluster_two:my-index//] | ||
|
|
||
| Which returns: | ||
|
|
||
| [source,console-result] | ||
| ---- | ||
| { | ||
| "is_running": false, | ||
| "took": 42, <1> | ||
| "columns" : [ | ||
| { | ||
| "name" : "COUNT(http.response.status_code)", | ||
| "type" : "long" | ||
| }, | ||
| { | ||
| "name" : "user.id", | ||
| "type" : "keyword" | ||
| } | ||
| ], | ||
| "values" : [ | ||
| [4, "elkbee"], | ||
| [1, "kimchy"] | ||
| ], | ||
| "_clusters": { <2> | ||
| "total": 3, | ||
| "successful": 3, | ||
| "running": 0, | ||
| "skipped": 0, | ||
| "partial": 0, | ||
| "failed": 0, | ||
| "details": { <3> | ||
| "(local)": { <4> | ||
| "status": "successful", | ||
| "indices": "blogs", | ||
| "took": 36, <5> | ||
| "_shards": { <6> | ||
| "total": 13, | ||
| "successful": 13, | ||
| "skipped": 0, | ||
| "failed": 0 | ||
| } | ||
| }, | ||
| "cluster_one": { | ||
| "status": "successful", | ||
| "indices": "cluster_one:my-index-000001", | ||
| "took": 38, | ||
| "_shards": { | ||
| "total": 4, | ||
| "successful": 4, | ||
| "skipped": 0, | ||
| "failed": 0 | ||
| } | ||
| }, | ||
| "cluster_two": { | ||
| "status": "successful", | ||
| "indices": "cluster_two:my-index-000001", <7> | ||
| "took": 41, | ||
| "_shards": { | ||
| "total": 18, | ||
| "successful": 18, | ||
| "skipped": 1, | ||
| "failed": 0 | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ---- | ||
| // TEST[skip: cross-cluster testing env not set up] | ||
|
|
||
| <1> How long the entire search (across all clusters) took, in milliseconds. | ||
| <2> This section of counters shows all possible cluster search states and how many cluster | ||
| searches are currently in that state. The clusters can have one of the following statuses: *running*, | ||
| *successful* (searches on all shards were successful), *skipped* (the search | ||
| failed on a cluster marked with `skip_unavailable`=`true`) or *failed* (the search | ||
| failed on a cluster marked with `skip_unavailable`=`false`). | ||
| <3> The `_clusters/details` section shows metadata about the search on each cluster. | ||
| <4> If you included indices from the local cluster you sent the request to in your {ccs}, | ||
| it is identified as "(local)". | ||
| <5> How long (in milliseconds) the search took on each cluster. This can be useful to determine | ||
| which clusters have slower response times than others. | ||
| <6> The shard details for the search on that cluster, including a count of shards that were | ||
| skipped due to the can-match phase indicating it had no matching data so it did not need | ||
| to be included in the full ES|QL query. | ||
| <7> The index expression supplied by the user. If you provide a wildcard such as `my-index*`, | ||
| this section will show the resolved index name(s) here, unless no matching indices could | ||
| be found on that cluster, in which case the wildcard expression will be retained here. | ||
|
|
||
|
|
||
| The cross-cluster metadata can be used to determine whether any data came back from a cluster. | ||
| For instance in the query below, you see that wildcard expression for `cluster-two` did not | ||
| resolve to a concrete index (or indices) and that the total number of shards searched is | ||
| zero. This indicates that no matching index was found on that cluster. But since the other | ||
| cluster did have a matching index, the search did not return an error, but instead | ||
| returned all the matching data it could find. | ||
|
|
||
|
|
||
| [source,console] | ||
| ---- | ||
| POST /_query/async?format=json | ||
| { | ||
| "query": """ | ||
| FROM cluster_one:my-index*,cluster_two:logs* | ||
| | STATS COUNT(http.response.status_code) BY user.id | ||
| | LIMIT 2 | ||
| """ | ||
| } | ||
| ---- | ||
| // TEST[continued] | ||
| // TEST[s/cluster_one:my-index\*,cluster_two:logs\*/my-index-000001/] | ||
|
|
||
| Which returns: | ||
|
|
||
| [source,console-result] | ||
| ---- | ||
| { | ||
| "is_running": false, | ||
| "took": 55, | ||
| "columns": [ | ||
| ... // not shown | ||
| ], | ||
| "values": [ | ||
| ... // not shown | ||
| ], | ||
| "_clusters": { | ||
| "total": 2, | ||
| "successful": 2, | ||
| "running": 0, | ||
| "skipped": 0, | ||
| "partial": 0, | ||
| "failed": 0, | ||
| "details": { | ||
| "cluster_one": { | ||
| "status": "successful", | ||
| "indices": "cluster_one:my-index-000001", | ||
| "took": 38, | ||
| "_shards": { | ||
| "total": 4, | ||
| "successful": 4, | ||
| "skipped": 0, | ||
| "failed": 0 | ||
| } | ||
| }, | ||
| "cluster_two": { | ||
| "status": "successful", <1> | ||
| "indices": "cluster_two:logs*", <2> | ||
| "took": 0, | ||
| "_shards": { | ||
| "total": 0, <3> | ||
| "successful": 0, | ||
| "skipped": 0, | ||
| "failed": 0 | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ---- | ||
| // TEST[skip: cross-cluster testing env not set up] | ||
|
|
||
| <1> This search is still marked as successful, even though no data was searched. | ||
| <2> Since there were no matching indices for the wildcard pattern provided, the original | ||
| index expression provided by the user is retained here. | ||
| <3> Indicates that no shards were searched (due to not having any matching indices). | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
There was a problem hiding this comment. @naj-h and @tylerperk - Please review proposed end user docs changes. |
||
| [discrete] | ||
| [[ccq-enrich]] | ||
| ==== Enrich across clusters | ||
|
|
@@ -331,8 +529,7 @@ setting. As a result, if a remote cluster specified in the request is | |
| unavailable or failed, {ccs} for {esql} queries will fail regardless of the setting. | ||
|
|
||
| We are actively working to align the behavior of {ccs} for {esql} with other | ||
| {ccs} APIs. | ||
|
|
||
| [discrete] | ||
| [[ccq-during-upgrade]] | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The local cluster should be available, right? Could we remove the multi-cluster output so we get the assertion that the shape is pretty close?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I spent several hours trying but unless you know of a trick to do clever multi-line matching I don't see how this is possible. Among the things I tried was adding "m" to the end of the matcher to indicate multi-line matching (as in Perl matching), but that doesn't work. Mostly I just get failed runs with no information as to what is wrong.
Plus I'm not really sure it's worth it? The whole point of this section is to show the
_clusters/detailssection so testing against a non-CCS set up doesn't seem useful.We probably need another ticket to enable the multi-cluster testing setup that search-across-clusters.asciidoc uses, as that was not set up for this test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍