Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fuzzy CI debugging #7

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Fuzzy CI debugging #7

wants to merge 8 commits into from

Conversation

pitag-ha
Copy link
Owner

No description provided.

@github-actions GitHub Actions
Copy link

This PR introduces a change in some Merlin query response(s), that's captured by the fuzzy-test CI. The diff of the CI has been approved in its following state:

  • 256-sha of full reponses diff hash: a61c137e7f8548c7502b438c62739005c0b70b660c342e3b6d724a32c1f98247
  • The diff can be found in the artifact diff-53eaad5df01bbdabc9e655dbadebc302d0475f31-512c9eead6c2755a4b6970a86ef35c4247a91bee on https://github.com/pitag-ha/merlin/actions/runs/6945284063 . It contains the difference in the responses of ocamlmerlin when run on randomly chosen samples
    • with merlin checked out to the PR target, i.e. master at 53eaad5df01bbdabc9e655dbadebc302d0475f31
    • with merlin checked out to the PR merge branch, i.e. merge result of fuzzy-ci at 512c9eead6c2755a4b6970a86ef35c4247a91bee and the target

Note: The responses on the samples have also been distilled down to

  • What kind of response does Merlin give (return or failure or error etc.)?
  • Has the Merlin server crashed since the last query?

However, the behavior change of this PR is too subtile to be caught by that distilled data. Have a look at the full responses.

@github-actions GitHub Actions
Copy link

This PR introduces a change in some Merlin query response(s), that's captured by the fuzzy-test CI. The diff of the CI has been approved in its following state:

  • 256-sha of full reponses diff hash: a61c137e7f8548c7502b438c62739005c0b70b660c342e3b6d724a32c1f98247
  • The diff can be found in the artifact diff-53eaad5df01bbdabc9e655dbadebc302d0475f31-3697753edfe2cb5433aabb6a5f3f4860c4f2be19 on https://github.com/pitag-ha/merlin/actions/runs/6945492544 . It contains the difference in the responses of ocamlmerlin when run on randomly chosen samples with Merlin checked out to 1. the PR's base branch and 2. the PRs merge branch.

Note: The responses on the samples have also been distilled down to

  • What kind of response does Merlin give (return or failure or error etc.)?
  • Has the Merlin server crashed since the last query?

Each of the three artifacts on https://github.com/pitag-ha/merlin/actions/runs/6945492544 contain both the distilled version and the full reponses.

@github-actions GitHub Actions
Copy link

This PR introduces a change in some Merlin query response(s), that's captured by the fuzzy-test CI. The diff of the CI has been approved in its following state:

  • 256-sha of full reponses diff hash: a61c137e7f8548c7502b438c62739005c0b70b660c342e3b6d724a32c1f98247
  • The diff can be found in the artifact diff-53eaad5df01bbdabc9e655dbadebc302d0475f31-dd530f3e3569f6b35782336bf6be6aaf0b6f69ed on https://github.com/pitag-ha/merlin/actions/runs/6946306907 . It contains the difference in the responses of ocamlmerlin when run on randomly chosen samples with Merlin checked out to 1. the PR's base branch and 2. the PRs merge branch.

Note: The responses on the samples have also been distilled down to

  • What kind of response does Merlin give (return or failure or error etc.)?
  • Has the Merlin server crashed since the last query?

Each of the three artifacts on https://github.com/pitag-ha/merlin/actions/runs/6946306907 contain both the distilled version and the full reponses.

@pitag-ha pitag-ha force-pushed the fuzzy-ci branch 5 times, most recently from ea3ac80 to a1b22c6 Compare November 22, 2023 12:39
@pitag-ha pitag-ha force-pushed the fuzzy-ci branch 5 times, most recently from f3c2918 to 9ca81dc Compare November 26, 2023 15:18
@pitag-ha pitag-ha force-pushed the fuzzy-ci branch 5 times, most recently from 88e8332 to 4a415eb Compare December 1, 2023 21:13
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2023

This PR changes the response of some of the queries, that were run and analyzed by the Merlin Fuzzy CI. The change is not considered a regression, the analyzis of this PR has been approved in its following state:

pitag-ha and others added 4 commits December 4, 2023 15:24
@pitag-ha
First POC with dummy data
4b37791 
First working POC

A few updates

Checking out GH action artefacts

Find common ancestor

Useless commit to test GH action

Create data in parallel, save as artefacts

Save shas in job output

Add diff step

Try sending a comment to the PR

Improve save shas workflow

Upload diff as artifact

Upload diff as artifact

Fix conditional and extract shared variables

Co-authored-by: Enguerrand Decorne <[email protected]>

Fix label logic conditional

Co-authored-by: Enguerrand Decorne <[email protected]>

Do some stuff

Co-authored-by: Enguerrand Decorne <[email protected]>

Fix the approval workflow and others

Restructure

Remove timestamp from diff

Follow up on approval workflow

Fix artifact download

Use curl to download diff artifact

Debug why the gh api rate limit gets exceeded

Fix the approve workflow, i.e. sending a long msg contianing hash etc

Use GITHUB_TOKEN again (works when using the right syntax)

Update comment msg

Add output logic

Debug comparison of diff hashes

Handle both full reponses data and distilled data

Try to be more coherent with quotes

Fix URLs

Improve env hygiene and remove artifact_names job

Improve comment on approval

Create real data

Update merl-an

Update base branch name

Fix diff name and diff output

Disable the other gh actions for now

Fix the approval msg
@pitag-ha @abbysmal
Document env vars and pull in Irmin
886edb3 
Co-authored-by: Enguerrand Decorne <[email protected]>

Document the env variables

Co-authored-by: Enguerrand Decorne <[email protected]>

Pull in Irmin

Co-authored-by: Enguerrand Decorne <[email protected]>
@pitag-ha
Create significant data and readable diff
73146b9 
Install Irmin deps and build Irmin

Improve the diff output

Update the permissions

Format the diffs in a readable way

Move Irmin's lockfile

Fix diff creation

Debug why ocamlopt isn't in the PATH

Create more data

Reduce dif context

Build test/irmin-pack as well

Make sure Irmin deps are solved from its lock file

Use opam-monorepo to pull Irmin's dependencies instead of opam

Debugging and first irmin build cache implementation

More debugging and first merl-an cache implementation

Try both irmin and merl-an cache

Clean up the PR comment and stdout msgs

Last polishing of the diff output

Avoid triggering the workflow when unnecessary

Clean up commented code and debug steps

Make create_diff script more readable

Clean up and format the yml-file
@pitag-ha
Make the CI run again
4fdcb47
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2023

This PR changes the response of some of the queries, that were run and analyzed by the Merlin Fuzzy CI. The change is not considered a regression, the analyzis of this PR has been approved in its following state:

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2023

This PR changes the response of some of the queries, that were run and analyzed by the Merlin Fuzzy CI. The change is not considered a regression, the analyzis of this PR has been approved in its following state:

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 4, 2023

This PR changes the response of some of the queries, that were run and analyzed by the Merlin Fuzzy CI. The change is not considered a regression, the analyzis of this PR has been approved in its following state:

@pitag-ha pitag-ha force-pushed the fuzzy-ci branch 3 times, most recently from 88995e4 to d31044a Compare December 7, 2023 13:50
@merlin-ois
Copy link

Hello :)

@merlin-ois
Copy link

Hello :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
fuzzy-diff-looks-good
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pitag-ha @merlin-ois