Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] MixedClusterClientYamlTestSuiteIT class failing #113730

Closed
elasticsearchmachine opened this issue Sep 27, 2024 · 7 comments · Fixed by #113759
Closed

[CI] MixedClusterClientYamlTestSuiteIT class failing #113730

elasticsearchmachine opened this issue Sep 27, 2024 · 7 comments · Fixed by #113759
Assignees
@dnhatn
Labels
needs:risk Requires assignment of a risk label (low, medium, blocker) :StorageEngine/Mapping The storage related side of mappings Team:StorageEngine >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

Build Scans:

Reproduction Line:

./gradlew ":qa:mixed-cluster:v8.16.0#mixedClusterTest" -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=aggregations/cardinality_metric/Metadata test}" -Dtests.seed=45190E2FADCE34E0 -Dtests.bwc=true -Dtests.locale=ksh-Latn-DE -Dtests.timezone=Portugal -Druntime.java=22

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [aggregations/cardinality_metric:2]: expected [2xx] status code but api [indices.create] returned [503 Service Unavailable] [{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null,"stack_trace":"org.elasticsearch.discovery.MasterNotDiscoveredException\n"}],"type":"master_not_discovered_exception","reason":null,"stack_trace":"org.elasticsearch.discovery.MasterNotDiscoveredException\n"},"status":503}]

Issue Reasons:

  • [main] 2 consecutive failures in class org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT
  • [main] 3 consecutive failures in step 8.16.0_bwc-snapshots
  • [main] 2 consecutive failures in pipeline elasticsearch-pull-request
  • [main] 74 failures in class org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT (10.9% fail rate in 680 executions)
  • [main] 69 failures in step 8.16.0_bwc-snapshots (11.6% fail rate in 596 executions)
  • [main] 4 failures in step 8.16.0_bwc (9.1% fail rate in 44 executions)
  • [main] 6 failures in pipeline elasticsearch-intake (3.0% fail rate in 201 executions)
  • [main] 63 failures in pipeline elasticsearch-pull-request (15.3% fail rate in 411 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Search Relevance/Vectors Vector search >test-failure Triaged test failures from CI labels Sep 27, 2024
elasticsearchmachine added a commit that referenced this issue Sep 27, 2024
@elasticsearchmachine
Mute org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT or…
20edb06 
…g.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT #113730
@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 2 consecutive failures in class org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT
  • [main] 3 consecutive failures in step 8.16.0_bwc-snapshots
  • [main] 2 consecutive failures in pipeline elasticsearch-pull-request
  • [main] 74 failures in class org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT (10.9% fail rate in 680 executions)
  • [main] 69 failures in step 8.16.0_bwc-snapshots (11.6% fail rate in 596 executions)
  • [main] 4 failures in step 8.16.0_bwc (9.1% fail rate in 44 executions)
  • [main] 6 failures in pipeline elasticsearch-intake (3.0% fail rate in 201 executions)
  • [main] 63 failures in pipeline elasticsearch-pull-request (15.3% fail rate in 411 executions)

Build Scans:

@elasticsearchmachine elasticsearchmachine added needs:risk Requires assignment of a risk label (low, medium, blocker) Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch labels Sep 27, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-search-relevance (Team:Search Relevance)

@benwtrent benwtrent added :Analytics/Aggregations Aggregations and removed :Search Relevance/Vectors Vector search labels Sep 28, 2024
@elasticsearchmachine elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) and removed Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch labels Sep 28, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-analytical-engine (Team:Analytics)

@iverase
Copy link
Contributor

iverase commented Sep 28, 2024

Looking at the error it looks more synthetic source related

» [2024-09-27T20:38:29,809][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v8.16.0-3] fatal error in thread [elasticsearch[v8.16.0-3][masterService#updateTask][T#3]], exiting java.lang.AssertionError: soft deletes must be enabled in version 8515000
»  	at [email protected]/org.elasticsearch.index.IndexSettings.<init>(IndexSettings.java:936)
»  	at [email protected]/org.elasticsearch.indices.IndicesService.createIndexMapperServiceForValidation(IndicesService.java:820)
»  	at org.elasticsearch.xpack.logsdb.SyntheticSourceIndexSettingsProvider.newIndexHasSyntheticSourceUsage(SyntheticSourceIndexSettingsProvider.java:80)
»  	at org.elasticsearch.xpack.logsdb.SyntheticSourceIndexSettingsProvider.getAdditionalIndexSettings(SyntheticSourceIndexSettingsProvider.java:59)
»  	at [email protected]/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.aggregateIndexSettings(MetadataCreateIndexService.java:1004)
»  	at [email protected]/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV1Templates(MetadataCreateIndexService.java:602)

@iverase iverase added :StorageEngine/Mapping The storage related side of mappings and removed :Analytics/Aggregations Aggregations labels Sep 28, 2024
@elasticsearchmachine elasticsearchmachine added Team:StorageEngine and removed Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) labels Sep 28, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-storage-engine (Team:StorageEngine)

@dnhatn
Copy link
Member

dnhatn commented Sep 28, 2024

This seems to be caused by #113522

@dnhatn dnhatn self-assigned this Sep 28, 2024
@dnhatn dnhatn mentioned this issue Sep 29, 2024
Merged
@dnhatn
Copy link
Member

dnhatn commented Sep 29, 2024

@martijnvg I looked at the test failures and I think there are potential issues with creating IndexSettings inside SyntheticSourceIndexSettingsProvider#getAdditionalIndexSettings, as the passed settings might be incomplete or unverified.

  • ./gradlew ":qa:mixed-cluster:v8.16.0#mixedClusterTest" -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=indices.split/40_routing_partition_size/nested}"

  • ./gradlew ":qa:mixed-cluster:v8.16.0#mixedClusterTest" -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=indices.create/10_basic/Create index without soft deletes}"

I've reverted #113522 and unmuted the MixedClusterClientYamlTestSuiteIT suite to avoid disrupting other work.

dnhatn added a commit that referenced this issue Sep 29, 2024
@dnhatn
Unmute MixedClusterClientYamlTestSuiteIT (#113743)
7cff27a 
Relates #113730
@martijnvg martijnvg mentioned this issue Sep 30, 2024
Merged
martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Oct 1, 2024
@martijnvg
Reapply "synthetic source index setting provider should check source …
ea8ce0f 
…field mapper" (elastic#113759)

Originally added via elastic#113522, but then reverted via elastic#113745, because of mixed cluster test failures (elastic#113730).

This PR is a clean revert of the commit the reverted elastic#113522 and one additional commit that should address the build failures report in elastic#113730 : c7bd242

Basically create index invocation that would fail anyway should be ignored. If mapper service creation now fails, then we just assume that there is no synthetic source usage. This is ok, because the index creation would fail anyway later one.

Closes elastic#113730
@martijnvg martijnvg mentioned this issue Oct 1, 2024
Merged
martijnvg added a commit that referenced this issue Oct 1, 2024
@martijnvg
[8.x] Reapply "synthetic source index setting provider should check s…
945aaa2 
…ource field mapper" (#113759) (#113828)

Backporting #113759 to 8.x branch.

Originally added via #113522, but then reverted via #113745, because of mixed cluster test failures (#113730).

This PR is a clean revert of the commit the reverted #113522 and one additional commit that should address the build failures report in #113730 : c7bd242

Basically create index invocation that would fail anyway should be ignored. If mapper service creation now fails, then we just assume that there is no synthetic source usage. This is ok, because the index creation would fail anyway later one.

Closes #113730
matthewabbott pushed a commit to matthewabbott/elasticsearch that referenced this issue Oct 4, 2024
@elasticsearchmachine @matthewabbott
Mute org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT or…
52da43f 
…g.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT elastic#113730
matthewabbott pushed a commit to matthewabbott/elasticsearch that referenced this issue Oct 4, 2024
@dnhatn @matthewabbott
Unmute MixedClusterClientYamlTestSuiteIT (elastic#113743)
56789ab 
Relates elastic#113730
matthewabbott pushed a commit to matthewabbott/elasticsearch that referenced this issue Oct 4, 2024
@martijnvg @matthewabbott
Reapply "synthetic source index setting provider should check source …
bf829f8 
…field mapper" (elastic#113759)

Originally added via elastic#113522, but then reverted via elastic#113745, because of mixed cluster test failures (elastic#113730).

This PR is a clean revert of the commit the reverted elastic#113522 and one additional commit that should address the build failures report in elastic#113730 : c7bd242

Basically create index invocation that would fail anyway should be ignored. If mapper service creation now fails, then we just assume that there is no synthetic source usage. This is ok, because the index creation would fail anyway later one.

Closes elastic#113730
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
needs:risk Requires assignment of a risk label (low, medium, blocker) :StorageEngine/Mapping The storage related side of mappings Team:StorageEngine >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
4 participants
@benwtrent @dnhatn @iverase @elasticsearchmachine