ES|QL Physical plan serialization can be reduced/removed #113809
Labels
:Analytics/ES|QL
AKA ESQL
Team:Analytics
Meta label for analytical engine team (ESQL/Aggs/Geo)
>tech debt
In ES|QL the query plan transmitted to the data nodes (serialized) is a physical plan containing a FragmentExec which in turn contains a Logical Plan. This means that only the higher level nodes in the physical plan need to be serialized. Currently we maintain a lot of serialization code, and unit tests for this code, but it can be considered to be dead code.
This issue was first noticed during the development of pushdown to lucene of sorting by distance. At first we added additional serialization to
sorts
in theEsQueryExec
class, and dealt with the daily merge conflicts from theTransportVersions
change. But then we realized that this data was never serialized, so we removed the support for allsorts
serialization, but in a way that did not require transport version changes (ie. always serialize an empty list, and when deserializing, ignore the results).However, on thinking more about this we realize that the entire
EsQueryExec
class itself might never be serialized. We potentially remove a lot of dead serialization code if many classes are never seriealized. We do need to verify the scope of this, and also take into account thepragma
node_level_reduction
which turns on and off how much of the plan is handled at the data node versus the coordinator node. It is possible that pragma prevents us from removing any serialization, or removing that pragma.The text was updated successfully, but these errors were encountered: