Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-run only failed tests (jobs) for Cloud parallel recording #1290

Open
MikeMcC399 opened this issue Oct 31, 2024 · 2 comments
Open

Re-run only failed tests (jobs) for Cloud parallel recording #1290

MikeMcC399 opened this issue Oct 31, 2024 · 2 comments
Labels

Comments

@MikeMcC399
Copy link
Collaborator

MikeMcC399 commented Oct 31, 2024

What would you like?

Scenario

A GitHub Actions workflow contains a job using the parallel option of cypress-io/github-action recording to Cypress Cloud which runs Cypress tests in multiple containers.

One or more Cypress tests fail.

GitHub Actions offers the option Re-run failed jobs

image

Enhancement

GitHub Actions and Cypress Cloud should work together so that selecting Re-run failed jobs only re-runs failed jobs using a frozen set of load-balanced Cypress test specs, so that each failed job runs with the same test specs it used in its previous run. The Cypress tests which were distributed to other jobs, and which were previously successful, should not be re-run.

Why is this needed?

Reduce the overhead of re-running flaky tests without also running tests which were previously successful.

Other

The README > parallel documentation section currently states:

If you use the GitHub Actions facility for Re-running workflows and jobs, note that Re-running failed jobs in a workflow is not suited for use with parallel recording into Cypress Cloud. Re-running failed jobs in this situation does not simply re-run failed Cypress tests. Instead it re-runs all Cypress tests, load-balanced over the containers with failed jobs.

To optimize runs when there are failing tests present, refer to optional Cypress Cloud Smart Orchestration Premium features:

@theomelo
Copy link

@MikeMcC399 I would like to investigate this issue. Can you confirm if a deep knowledge of Cypress Cloud's "inner workings" (how the parallelization works, etc.) is required?

@MikeMcC399
Copy link
Collaborator Author

MikeMcC399 commented Nov 1, 2024

@theomelo

Can you confirm if a deep knowledge of Cypress Cloud's "inner workings" (how the parallelization works, etc.) is required?

I can't help you with your question. https://docs.cypress.io/cloud/features/smart-orchestration/parallelization describes the parallel load-sharing process. Although the Cypress code on https://github.com/cypress-io/cypress is open source and online viewable, the Cypress Cloud code, which implements load-sharing, is proprietary to Cypress.io and not shared online. I don't have access to the Cypress Cloud code myself and so I can't say how much you would need to know about it.

@MikeMcC399 MikeMcC399 changed the title Re-run only failed tests for Cloud parallel recording Re-run only failed tests (jobs) for Cloud parallel recording Nov 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants