Collect and display execution metadata for ES|QL cross cluster searches #112595
Commits on Sep 13, 2024
-
Collect and display execution metadata for ES|QL cross cluster searches
Added EsqlExecutionInfo class and added it to the EsqlQueryResponse, but not yet in the XContent output of EsqlQueryResponse. Also added to TransportEsqlQueryAction, but not yet passed down to the execution layers.
-
-
Removed changes to EsqlQueryResponse, spending too long getting the E…
…sqlQueryResponseTests to pass and probably don't want it in the response for now anyway
-
Starting threading EsqlExecutionInfo into PlanExecutor and EsqlSesssi…
…on, but not used yet to record metadata
-
Have the initial swap-in of cluster info into EsqlExecutionInfo in Es…
…qlSession.preAnalyzeIndices. No updates after that added yet. Passing in Function<String, Map<String, OriginalIndices>> rather than RemoteClusterService to PlanExecutor and EsqlSession. Makes it easier (or possible) to get the existing unit tests working again and less overall coupling.
-
Added EsqlExecutionInfo to IndexResolver. Enrich pathway passes in nu…
…ll for EsqlExecutionInfo. The EsqlExecutionInfo Cluster statuses are update in IndexResolver.updateExecutionInfoWithFieldCapsResults. Test for that method added.
-
ComputeListener updated to the version that has proper remote/local s…
…plit and new static constructors. However, the new acquire fns in ComputeListener are not yet used in ComputeService. ComputeResponse updated to have info (shard accounting and took time) need for adding to EsqlExecutionInfo, but again not being used/added yet. Added to queryStartTimeMillis to esql.session.Configuration
-
Added new tests to ComputeListenerTests
And adjusted acquireComputeForDataNodes to take long for startTime rather than whole Configuration object
-
update ExecutionInfo with shard counts in ComputeService.lookupDataNodes
Added new DataNodeResult record to be returned from ComputeService.lookupDataNodes
-
Migrated CrossClustersQueryIT to new setup format, but can't add exec…
…Info tests until its added to the EsqlQueryResponse
-
Added CountDown to acquireComputeForDataNodes - that allows SUCCESSFU…
…L status to be set for local cluster when last node finishes Added EsqlExecutionInfo to EsqlQueryResponse, but only took time is in the XContent Added new tests and test cases to CrossClustersQueryIT
-
Fixed failing REST and qa tests to account for the new 'took' time in…
… the ESQL response
-
-
-
Updated end user docs with info about top level took time and _cluste…
…r/details section
-
Added EsqlExecutionInfo to equals and hashCode method of EsqlQueryRes…
…ponse and got EsqlQueryResponseTests passing
-
Removed skip_unavilable=true filter in IndexResolver - all clusters a…
…re marked as skipped for now if remote cannot be contacted
-
-
Added equals and hashCode to EsqlExecutionInfo.Cluster object.
Improved javadoc and fixed a few minor test flaws. Removed some debugging code.
-
-
-
-
Added questions about took time headers to EsqlResponseListener - pos…
…sibly need to change that impl
Commits on Sep 16, 2024
Commits on Sep 17, 2024
Commits on Sep 18, 2024
-
PR feedback with focus on end user docs fixes, removing some out-of-d…
…ate code comments and minor improvements to tests (qa and IT)
Commits on Sep 19, 2024
-
Additional PR feedback changes - test adjustments, remove 'set' and '…
…get' prefixes from some methdos in EsqlExecutionInfo
-
Now tracking took in nanos, not millis (but XContent still displays i…
…n millis, until a decision is made to change it) Also reading/writing TimeValue with read/writeOptionalTimeValue.
-
EsqlResponseListener now preferentially uses the took time in the Esq…
…lExecutionInfo; falls back to internal Stopwatch
Commits on Sep 20, 2024
-
Modified esql-across-clusters to run the new queries I added; but JSO…
…N responses are not tested - not clear how to do multiline replacements in this test framework
Commits on Sep 23, 2024
-
Removed code that lists fully resolved indices in the _clusters/detai…
…ls (PR feedback) and updated tests
Commits on Sep 24, 2024
-
PR feedback: Moved logic for unavailable/missing clusters to EsqlSession
EsqlExecutionInfo is no longer passed into IndexResolver. Instead it returns the info needed in IndexResolution so that ExecInfo.Cluster info can be updated in EsqlSession.
Commits on Sep 26, 2024
-
PR feedback: I removed acquireCCSCompute and acquireComputeForDatanod…
…es in ComputeListener. This however requires adding additional state to ComputeListener to know what "context" each compute listener is running so it knows what actions to take.
-
Merge remote-tracking branch 'elastic/main' into esql/ccs-execution-i…
…nfo2-26-Sep-SQUASHED
-
PR feedback: Created new intf IndicesExpressionResolver and have Remo…
…teClusterService implementing it
Commits on Sep 27, 2024
-
-
Moved parseClusterAlias from IndexResolver to RemoteClusterAware and …
…using the new methods in RemoteClusterAware
-
Renamed IndicesExpressionResolver intf to IndicesExpressionGrouper.
Added ExecutionInfo asserts to CrossClustersEnrichIT.
-
PR feedback: Added javadoc to ComputeListener, removed leftover debug…
…ging code, other minor code cleanup.
-
Fixed bug where SKIPPED status for unavailable clusters from field-ca…
…ps lookup was being overwritten to SUCCESSFUL status