Collect and display execution metadata for ES|QL cross cluster searches #112595
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| pr: 112595 | ||
| summary: Collect and display execution metadata for ES|QL cross cluster searches | ||
| area: ES|QL | ||
| type: enhancement | ||
| issues: | ||
| - 112402 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -85,7 +85,7 @@ POST /_security/role/remote1 | |
| "privileges": [ "read","read_cross_cluster" ], <4> | ||
| "clusters" : ["my_remote_cluster"] <5> | ||
| } | ||
| ], | ||
| "remote_cluster": [ <6> | ||
| { | ||
| "privileges": [ | ||
|
|
@@ -174,6 +174,184 @@ FROM *:my-index-000001 | |
| | LIMIT 10 | ||
| ---- | ||
|
|
||
| [discrete] | ||
| [[ccq-cluster-details]] | ||
| ==== Cross-cluster metadata | ||
|
|
||
| ES|QL {ccs} responses include metadata about the search on each cluster when the response format is JSON. | ||
| Here we show an example using the async search endpoint. {ccs-cap} metadata is also present in the synchronous | ||
| search endpoint. | ||
|
|
||
| [source,esql] | ||
| ---- | ||
| POST /_query/async?format=json | ||
| { | ||
| "query": """ | ||
| FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index* | ||
| | KEEP author, name, page_count | ||
| | SORT page_count DESC | ||
| | LIMIT 50 | ||
| """ | ||
| } | ||
| ---- | ||
|
|
||
| Which returns: | ||
|
|
||
| [source,console-result] | ||
| ---- | ||
| { | ||
| "is_running": false, | ||
| "took": 42, <1> | ||
| "columns": [ | ||
| ... // not shown | ||
| ], | ||
| "values": [ | ||
| ... // not shown | ||
| ], | ||
| "_clusters": { <2> | ||
| "total": 3, | ||
| "successful": 3, | ||
| "running": 0, | ||
| "skipped": 0, | ||
| "partial": 0, | ||
| "failed": 0, | ||
| "details": { <3> | ||
| "(local)": { <4> | ||
| "status": "successful", | ||
| "indices": "blogs", | ||
| "took": 36, <5> | ||
| "_shards": { <6> | ||
| "total": 13, | ||
| "successful": 13, | ||
| "skipped": 0, | ||
| "failed": 0 | ||
| } | ||
| }, | ||
| "cluster_one": { | ||
| "status": "successful", | ||
| "indices": "cluster_one:my-index-000001", | ||
| "took": 38, | ||
| "_shards": { | ||
| "total": 4, | ||
| "successful": 4, | ||
| "skipped": 0, | ||
| "failed": 0 | ||
| } | ||
| }, | ||
| "cluster_two": { | ||
| "status": "successful", | ||
| "indices": "cluster_two:my-index-000001", <7> | ||
| "took": 41, | ||
| "_shards": { | ||
| "total": 18, | ||
| "successful": 18, | ||
| "skipped": 1, | ||
| "failed": 0 | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ---- | ||
| // TEST[skip: cross-cluster testing env not set up] | ||
|
There was a problem hiding this comment. The local cluster should be available, right? Could we remove the multi-cluster output so we get the assertion that the shape is pretty close? There was a problem hiding this comment. I spent several hours trying but unless you know of a trick to do clever multi-line matching I don't see how this is possible. Among the things I tried was adding "m" to the end of the matcher to indicate multi-line matching (as in Perl matching), but that doesn't work. Mostly I just get failed runs with no information as to what is wrong. Plus I'm not really sure it's worth it? The whole point of this section is to show the We probably need another ticket to enable the multi-cluster testing setup that search-across-clusters.asciidoc uses, as that was not set up for this test. |
||
|
|
||
| <1> How the long the entire search (across all clusters) took, in milliseconds. | ||
quux00 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| <2> This section of counters shows all possible cluster search states and how many cluster | ||
| searches are currently in that state. The clusters can be one of the following statuses: *running*, | ||
quux00 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| *successful* (searches on all shards were successful), *partial* (searches on at least | ||
| one shard of the cluster was successful and at least one failed), *skipped* (the search | ||
| failed on a cluster marked with `skip_unavailable`=`true`) or *failed* (the search | ||
| failed on a cluster marked with `skip_unavailable`=`false`). | ||
| <3> The `_clusters/details` section shows metadata about the search on each cluster. | ||
| <4> If you included indices from the local cluster you sent the request to in your {ccs}, | ||
| it is identified as "(local)". | ||
| <5> How long (in milliseconds) the search took on each cluster. This can be useful to determine | ||
| which clusters have slower response times than others. | ||
| <6> The shard details for the search on that cluster, including a count of shards that were | ||
| skipped due to the can-match phase indicating it had no matching data so it did not need | ||
| to be included in the full ESQL query. | ||
quux00 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| <7> The index expression supplied by the user. If you provide a wildcard such as `my-index*`, | ||
| this section will show the resolved index name(s) here, unless no matching indices could | ||
| be found on that cluster, in which case the wildcard expression will be retained here. | ||
|
|
||
|
|
||
| The cross-cluster metadata can be used to determine if any data came back from a cluster. | ||
| For instance in this query, you see that wildcard expression for `cluster-one` did not | ||
| resolve to a concrete index (or indices) and that the total number of shards searched is | ||
| zero. This indicates that no matching index was found on that cluster. But since the other | ||
| cluster did have a matching index, the search did not return an error, but instead | ||
| returned all the matching data it could find. | ||
|
|
||
| [source,esql] | ||
| ---- | ||
| POST /_query/async?format=json | ||
| { | ||
| "query": """ | ||
| FROM cluster_one:my-index*,cluster_two:logs* | ||
| | KEEP author, name, page_count | ||
| | SORT page_count DESC | ||
| | LIMIT 5 | ||
| """ | ||
| } | ||
| ---- | ||
|
|
||
| Which returns: | ||
|
|
||
| [source,console-result] | ||
| ---- | ||
| { | ||
| "is_running": false, | ||
| "took": 55, | ||
| "columns": [ | ||
| ... // not shown | ||
| ], | ||
| "values": [ | ||
| ... // not shown | ||
| ], | ||
| "_clusters": { | ||
| "total": 2, | ||
| "successful": 2, | ||
| "running": 0, | ||
| "skipped": 0, | ||
| "partial": 0, | ||
| "failed": 0, | ||
| "details": { | ||
| "cluster_one": { | ||
| "status": "successful", | ||
| "indices": "cluster_one:my-index-000001", | ||
| "took": 38, | ||
| "_shards": { | ||
| "total": 4, | ||
| "successful": 4, | ||
| "skipped": 0, | ||
| "failed": 0 | ||
| } | ||
| }, | ||
| "cluster_two": { | ||
| "status": "successful", <1> | ||
| "indices": "cluster_two:logs*", <2> | ||
| "took": 0, | ||
| "_shards": { | ||
| "total": 0, <3> | ||
| "successful": 0, | ||
| "skipped": 0, | ||
| "failed": 0 | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ---- | ||
| // TEST[skip: cross-cluster testing env not set up] | ||
|
|
||
| <1> This search is still marked as successful, even though no data was searched. | ||
| <2> Since there were no matching indices for the wildcard pattern provided, the original | ||
| index expression provided by the user is retained here. | ||
| <3> Indicates that no shards were searched (due to not having any matching indices). | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
There was a problem hiding this comment. @naj-h and @tylerperk - Please review proposed end user docs changes. |
||
| [discrete] | ||
| [[ccq-enrich]] | ||
| ==== Enrich across clusters | ||
|
|
@@ -331,8 +509,7 @@ setting. As a result, if a remote cluster specified in the request is | |
| unavailable or failed, {ccs} for {esql} queries will fail regardless of the setting. | ||
|
|
||
| We are actively working to align the behavior of {ccs} for {esql} with other | ||
| {ccs} APIs. | ||
|
|
||
| [discrete] | ||
| [[ccq-during-upgrade]] | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you make this a
consolesnippet instead and make the test runner happy with it with something like// TEST[setup:my_index]. That way we run it so if it ever goes out of date we fail the build.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess we'd have to do
// TEST[s/cluster_one:my-index-000001,cluster_two:my-index//]to remove the multi-cluster from the test case.....There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I have changed the two queries I added to actually be executed, but (see comment below) I am unable to find a way to test the response JSON, so I have left that as "TEST[skip...]".