Extra value returned in search response when sort fields are specified along with PIT.

[kiljag]

Elasticsearch Version

8.13.4

Installed Plugins

No response

Java Version

bundled

OS Version

Ubuntu

Problem Description

Hi,
When I try to sort the index with PIT, there is an extra value in sort field of the hit, where as the same is not present when the index is not queried without PIT.

Please let me know if this is expected or a bug.
Thank you.

Steps to Reproduce

Create a sample index and add few documents

PUT sample_index
{
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date"
      }
    }
  }
}

POST sample_index/_create/1
{
  "id": 1, "timestamp": "2024-10-21"
}

POST sample_index/_create/2
{
  "id": 2, "timestamp": "2024-10-22"
}

Search with sort fields without PIT

GET sample_index/_search
{
  
    "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    },
    {
      "id": {
        "order": "desc"
      }
    }
  ]
}

Response

"hits": [
      {
        "_index": "sample_index",
        "_id": "2",
        "_score": null,
        "_source": {
          "id": 2,
          "timestamp": "2024-10-22"
        },
        "sort": [
          1729555200000,
          2
        ]
      },
      {
        "_index": "sample_index",
        "_id": "1",
        "_score": null,
        "_source": {
          "id": 1,
          "timestamp": "2024-10-21"
        },
        "sort": [
          1729468800000,
          1
        ]
      }
    ]

Search with sort fields along with PIT

POST sample_index/_pit?keep_alive=1m

GET /_search
{
  "pit": {
    "id": "ucyMBAEMc2FtcGxlX2luZGV4Fl9KMzhxN1NrUUVLb0NJMF9FVUY0UFEAFjREbjM3VlVqUl9xX2xDU05NNGVWZkEAAAAAAABHwAwWOXlNbERIZXlRSldCRTZJX1R5MUxtZwABFl9KMzhxN1NrUUVLb0NJMF9FVUY0UFEAAA==",
    "keep_alive": "1m"
  }, 
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    },
    {
      "id": {
        "order": "desc"
      }
    }
  ]
}

Response

"hits": [
      {
        "_index": "sample_index",
        "_id": "2",
        "_score": null,
        "_source": {
          "id": 2,
          "timestamp": "2024-10-22"
        },
        "sort": [
          1729555200000,
          2,
          1 <== EXTRA_FIELD
        ]
      },
      {
        "_index": "sample_index",
        "_id": "1",
        "_score": null,
        "_source": {
          "id": 1,
          "timestamp": "2024-10-21"
        },
        "sort": [
          1729468800000,
          1,
          0 <== EXTRA_FIELD
        ]
      }
    ]

When the index is sorted using PIT, there is an extra field at 3rd index, where as the same is not present when simple sorting is applied. Is this expected?

Thank you.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[smalyshev]

I think the extra value is _shard_doc, from here: https://www.elastic.co/guide/en/elasticsearch/reference/8.15/paginate-search-results.html

All PIT search requests add an implicit sort tiebreaker field called _shard_doc, which can also be provided explicitly. If you cannot use a PIT, we recommend that you include a tiebreaker field in your sort. This tiebreaker field should contain a unique value for each document. If you don’t include a tiebreaker field, your paged results could miss or duplicate hits.