#101193 Preserve Step Info Across ILM Auto Retries

[lukewhiting]

This PR does a few things:

• Adds a new field to the Explain ILM lifecycle endpoint (<target>/_ilm/explain) called previous_step_info that contains the previous step runs step_info field contents
• Logic in the Lifecycle step transition logic to copy the step info of a previous run into the new field
• Docs for the new field

Bonus content includes:

• Refactored and simplified test object mutation logic
• Refactored test's to use recommended Clock interface for time dependent tests rather than custom supplier Reverted this as it's too risky / unrelated to PR

Commits are atomic so feel free to review commit by commit if you prefer reviewing smaller chunks

[github-actions]

Documentation preview:

• ✨ Changed pages

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Hi @lukewhiting, I've created a changelog YAML for you.

[elasticsearchmachine]

Hi @lukewhiting, I've updated the changelog YAML for you.

[lukewhiting]

Don't forget to close the issue in the PR description.

Yep :-) I have linked it to this PR so GitHub should auto close it once it's merged but I'll make a note to double check

[lukewhiting]

Local functional testing complete.

Method:

1. Index template created with policy that has a rollover step
2. Index (ilm-error-test-000001) created to match template and dummy doc inserted
3. Wait for ILM explain on index to confirm in check-rollover-ready state
4. Create new index (ilm-error-test-000000) that also matches the template
5. Manually create alias (ilm-error-test) for second index that matches rollover_alias of policy
6. ILM explain for the first index enters Error state with step info index.lifecycle.rollover_alias [ilm-error-test] does not point to index [ilm-error-test-000001]
7. Wait for auto retry
8. ILM explain for ilm-error-test-000001 shows following output that matches expected with failed_step_retry_count incremented by 1 and previous_step_info populated with previous attempts step info.

Test Output:

{
  "indices": {
    "ilm-error-test-000001": {
      "index": "ilm-error-test-000001",
      "managed": true,
      "policy": "365-days-default",
      "index_creation_date_millis": 1727179631766,
      "time_since_index_creation": "19.92m",
      "lifecycle_date_millis": 1727179631766,
      "age": "19.92m",
      "phase": "hot",
      "phase_time_millis": 1727180790334,
      "action": "rollover",
      "action_time_millis": 1727179631820,
      "step": "check-rollover-ready",
      "step_time_millis": 1727180790334,
      "is_auto_retryable_error": true,
      "failed_step_retry_count": 1,
      "previous_step_info": {
        "type": "illegal_argument_exception",
        "reason": "index.lifecycle.rollover_alias [ilm-error-test] does not point to index [ilm-error-test-000001]"
      },
      "phase_execution": {
        "policy": "365-days-default",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_docs": 200000000,
              "min_docs": 1,
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "version": 1,
        "modified_date_in_millis": 1727179591769
      }
    }
  }
}

[nielsbauman]

@lukewhiting That functional test looks great! Well done! I think it would also be worthwhile to ensure the previous_step_info isn't cleared after the second retry, but we can do that in the integration test that we'll be writing.

[gmarouli]

@lukewhiting nice work coming up with the test scenario!

Local functional testing complete.

Method:

1. Index template created with policy that has a rollover step
2. Index (ilm-error-test-000001) created to match template and dummy doc inserted
3. Wait for ILM explain on index to confirm in check-rollover-ready state
4. Create new index (ilm-error-test-000000) that also matches the template
5. Manually create alias (ilm-error-test) for second index that matches rollover_alias of policy
6. ILM explain for the first index enters Error state with step info index.lifecycle.rollover_alias [ilm-error-test] does not point to index [ilm-error-test-000001]
7. Wait for auto retry
8. ILM explain for ilm-error-test-000001 shows following output that matches expected with failed_step_retry_count incremented by 1 and previous_step_info populated with previous attempts step info.

Test Output:

{
  "indices": {
    "ilm-error-test-000001": {
      "index": "ilm-error-test-000001",
      "managed": true,
      "policy": "365-days-default",
      "index_creation_date_millis": 1727179631766,
      "time_since_index_creation": "19.92m",
      "lifecycle_date_millis": 1727179631766,
      "age": "19.92m",
      "phase": "hot",
      "phase_time_millis": 1727180790334,
      "action": "rollover",
      "action_time_millis": 1727179631820,
      "step": "check-rollover-ready",
      "step_time_millis": 1727180790334,
      "is_auto_retryable_error": true,
      "failed_step_retry_count": 1,
      "previous_step_info": {
        "type": "illegal_argument_exception",
        "reason": "index.lifecycle.rollover_alias [ilm-error-test] does not point to index [ilm-error-test-000001]"
      },
      "phase_execution": {
        "policy": "365-days-default",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_primary_shard_docs": 200000000,
              "min_docs": 1,
              "max_primary_shard_size": "50gb"
            }
          }
        },
        "version": 1,
        "modified_date_in_millis": 1727179591769
      }
    }
  }
}

I think this could be simplified in an java rest test for example:

1. Decrease ILM polling frequency to reduce the test duration
2. Index template created without the rollover alias but with a regular alias and with policy that has a rollover step with max_doc: 1
3. Index (ilm-error-test-000001) created to match template and dummy doc inserted
4. Wait for ILM explain on index to confirm in check-rollover-ready state
5. ILM explain will throw the error setting [index.lifecycle.rollover_alias] for index
6. Wait for auto retry
7. ILM explain for ilm-error-test-000001 shows following output that matches expected with failed_step_retry_count incremented by 1 and previous_step_info populated with previous attempts step info.

It's very similar with what you have just a few less steps. Let me know if you want to work on it together.

Nit: I think the test-update-serverless can be removed because ILM is not used in serverless. That should speed up ci as well.

[lukewhiting]

@nielsbauman @gmarouli Integration test for new field added based on the above manual testing: 91dac7e

[gmarouli]

Left a minor comment but LGTM :)

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.x