Self hosted runner: update image and make configurable through env vars #60
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rimrul
reviewed
Nov 22, 2023
There was a problem hiding this comment.
3 hours might be cutting it close-ish if we get a badly timed run.
This run was 2:32:22 (though it looks like an hour of that was waiting for a runner).
This run was 2:21:40.
This run was 2:15:05.
This run was 2:06:04.
This run was 16:42:42 (over 16h of this run where seemingly spent waiting for a runner).
The 16 hour one is clearly a very broken outlier, but for the clang runs it looks like the trend is going towards 2.5 hours of runtime and that might still grow longer.
We could start with 3 hours and increase the timout if we run into issues.
dscho
requested changes
Nov 22, 2023
There was a problem hiding this comment.
I do like the updates with regards to the image, but would want the logic to deal with VM spin-up problems elsewhere: in GitForWindowsHelper. More details below.
|
|
||
| on: | ||
| schedule: | ||
| # Run every 6 hours |
There was a problem hiding this comment.
Hmm. I don't really like to run this every 6 hours, when we rarely kick off runs. Nor do I like the idea of assuming that VMs that are older than 3 hours need to be decommissioned; Even if it is a long time for a workflow job to run, I would hate for a legitimate, slow-running job to be killed before finishing.
I would be more a fan of reacting to events. In response to the failed job referenced in #59, the GitForWindowsHelper received a check_run webhook event with the action set to completed and check_run.conclusion set to failure. This information can be used to react specifically to similar failures when spinning up a VM with a self-hosted runner.
The full webhook event sent to the GitForWindowsHelper GitHub App
{
"headers": {
"Accept": "*/*",
"Content-Type": "application/json",
"User-Agent": "GitHub-Hookshot/24a4d61",
"X-GitHub-Delivery": "f7e0eb00-885e-11ee-9c57-b83bb0090ef7",
"X-GitHub-Event": "check_run",
"X-GitHub-Hook-ID": "392499201",
"X-GitHub-Hook-Installation-Target-ID": "246505",
"X-GitHub-Hook-Installation-Target-Type": "integration",
"X-Hub-Signature": "sha1=424cf5379364ea1ca9e54397111cf67a7a104889",
"X-Hub-Signature-256": "sha256=52ac5f67ffc33e72d5319a25645f2f648b022bd3cf7165a096e854e35371bc7c"
},
"payload": {
"action": "completed",
"check_run": {
"id": 18885592052,
"name": "create-runner",
"node_id": "CR_kwDOIhGD8s8AAAAEZatD9A",
"head_sha": "62c7234114b0ff2590455c0757ab196f0b977dc2",
"external_id": "3220e755-65ba-54d3-a1cd-a9396bbeda13",
"url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/check-runs/18885592052",
"html_url": "https://github.com/git-for-windows/git-for-windows-automation/actions/runs/6942503689/job/18885592052",
"details_url": "https://github.com/git-for-windows/git-for-windows-automation/actions/runs/6942503689/job/18885592052",
"status": "completed",
"conclusion": "failure",
"started_at": "2023-11-21T10:32:01Z",
"completed_at": "2023-11-21T11:13:14Z",
"output": {
"title": null,
"summary": null,
"text": null,
"annotations_count": 2,
"annotations_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/check-runs/18885592052/annotations"
},
"check_suite": {
"id": 18372063685,
"node_id": "CS_kwDOIhGD8s8AAAAERw9xxQ",
"head_branch": "main",
"head_sha": "62c7234114b0ff2590455c0757ab196f0b977dc2",
"status": "completed",
"conclusion": "failure",
"url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/check-suites/18372063685",
"before": "10edc10b4cffc77ab7ada4c0466de6224debc2dc",
"after": "62c7234114b0ff2590455c0757ab196f0b977dc2",
"pull_requests": [
{
"url": "https://api.github.com/repos/frikke/git-for-windows-automation/pulls/1",
"id": 1425431422,
"number": 1,
"head": {
"ref": "main",
"sha": "62c7234114b0ff2590455c0757ab196f0b977dc2",
"repo": {
"id": 571573234,
"url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation",
"name": "git-for-windows-automation"
}
},
"base": {
"ref": "main",
"sha": "65a23e20ee99a88843c987a04375857c6f812669",
"repo": {
"id": 645542833,
"url": "https://api.github.com/repos/frikke/git-for-windows-automation",
"name": "git-for-windows-automation"
}
}
}
],
"app": {
"id": 15368,
"slug": "github-actions",
"node_id": "MDM6QXBwMTUzNjg=",
"owner": {
"login": "github",
"id": 9919,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjk5MTk=",
"avatar_url": "https://avatars.githubusercontent.com/u/9919?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/github",
"html_url": "https://github.com/github",
"followers_url": "https://api.github.com/users/github/followers",
"following_url": "https://api.github.com/users/github/following{/other_user}",
"gists_url": "https://api.github.com/users/github/gists{/gist_id}",
"starred_url": "https://api.github.com/users/github/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/github/subscriptions",
"organizations_url": "https://api.github.com/users/github/orgs",
"repos_url": "https://api.github.com/users/github/repos",
"events_url": "https://api.github.com/users/github/events{/privacy}",
"received_events_url": "https://api.github.com/users/github/received_events",
"type": "Organization",
"site_admin": false
},
"name": "GitHub Actions",
"description": "Automate your workflow from idea to production",
"external_url": "https://help.github.com/en/actions",
"html_url": "https://github.com/apps/github-actions",
"created_at": "2018-07-30T09:30:17Z",
"updated_at": "2019-12-10T19:04:12Z",
"permissions": {
"actions": "write",
"administration": "read",
"checks": "write",
"contents": "write",
"deployments": "write",
"discussions": "write",
"issues": "write",
"merge_queues": "write",
"metadata": "read",
"packages": "write",
"pages": "write",
"pull_requests": "write",
"repository_hooks": "write",
"repository_projects": "write",
"security_events": "write",
"statuses": "write",
"vulnerability_alerts": "read"
},
"events": [
"branch_protection_rule",
"check_run",
"check_suite",
"create",
"delete",
"deployment",
"deployment_status",
"discussion",
"discussion_comment",
"fork",
"gollum",
"issues",
"issue_comment",
"label",
"merge_group",
"milestone",
"page_build",
"project",
"project_card",
"project_column",
"public",
"pull_request",
"pull_request_review",
"pull_request_review_comment",
"push",
"registry_package",
"release",
"repository",
"repository_dispatch",
"status",
"watch",
"workflow_dispatch",
"workflow_run"
]
},
"created_at": "2023-11-21T10:31:52Z",
"updated_at": "2023-11-21T11:13:16Z"
},
"app": {
"id": 15368,
"slug": "github-actions",
"node_id": "MDM6QXBwMTUzNjg=",
"owner": {
"login": "github",
"id": 9919,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjk5MTk=",
"avatar_url": "https://avatars.githubusercontent.com/u/9919?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/github",
"html_url": "https://github.com/github",
"followers_url": "https://api.github.com/users/github/followers",
"following_url": "https://api.github.com/users/github/following{/other_user}",
"gists_url": "https://api.github.com/users/github/gists{/gist_id}",
"starred_url": "https://api.github.com/users/github/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/github/subscriptions",
"organizations_url": "https://api.github.com/users/github/orgs",
"repos_url": "https://api.github.com/users/github/repos",
"events_url": "https://api.github.com/users/github/events{/privacy}",
"received_events_url": "https://api.github.com/users/github/received_events",
"type": "Organization",
"site_admin": false
},
"name": "GitHub Actions",
"description": "Automate your workflow from idea to production",
"external_url": "https://help.github.com/en/actions",
"html_url": "https://github.com/apps/github-actions",
"created_at": "2018-07-30T09:30:17Z",
"updated_at": "2019-12-10T19:04:12Z",
"permissions": {
"actions": "write",
"administration": "read",
"checks": "write",
"contents": "write",
"deployments": "write",
"discussions": "write",
"issues": "write",
"merge_queues": "write",
"metadata": "read",
"packages": "write",
"pages": "write",
"pull_requests": "write",
"repository_hooks": "write",
"repository_projects": "write",
"security_events": "write",
"statuses": "write",
"vulnerability_alerts": "read"
},
"events": [
"branch_protection_rule",
"check_run",
"check_suite",
"create",
"delete",
"deployment",
"deployment_status",
"discussion",
"discussion_comment",
"fork",
"gollum",
"issues",
"issue_comment",
"label",
"merge_group",
"milestone",
"page_build",
"project",
"project_card",
"project_column",
"public",
"pull_request",
"pull_request_review",
"pull_request_review_comment",
"push",
"registry_package",
"release",
"repository",
"repository_dispatch",
"status",
"watch",
"workflow_dispatch",
"workflow_run"
]
},
"pull_requests": [
{
"url": "https://api.github.com/repos/frikke/git-for-windows-automation/pulls/1",
"id": 1425431422,
"number": 1,
"head": {
"ref": "main",
"sha": "62c7234114b0ff2590455c0757ab196f0b977dc2",
"repo": {
"id": 571573234,
"url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation",
"name": "git-for-windows-automation"
}
},
"base": {
"ref": "main",
"sha": "65a23e20ee99a88843c987a04375857c6f812669",
"repo": {
"id": 645542833,
"url": "https://api.github.com/repos/frikke/git-for-windows-automation",
"name": "git-for-windows-automation"
}
}
}
]
},
"repository": {
"id": 571573234,
"node_id": "R_kgDOIhGD8g",
"name": "git-for-windows-automation",
"full_name": "git-for-windows/git-for-windows-automation",
"private": false,
"owner": {
"login": "git-for-windows",
"id": 4571183,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjQ1NzExODM=",
"avatar_url": "https://avatars.githubusercontent.com/u/4571183?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/git-for-windows",
"html_url": "https://github.com/git-for-windows",
"followers_url": "https://api.github.com/users/git-for-windows/followers",
"following_url": "https://api.github.com/users/git-for-windows/following{/other_user}",
"gists_url": "https://api.github.com/users/git-for-windows/gists{/gist_id}",
"starred_url": "https://api.github.com/users/git-for-windows/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/git-for-windows/subscriptions",
"organizations_url": "https://api.github.com/users/git-for-windows/orgs",
"repos_url": "https://api.github.com/users/git-for-windows/repos",
"events_url": "https://api.github.com/users/git-for-windows/events{/privacy}",
"received_events_url": "https://api.github.com/users/git-for-windows/received_events",
"type": "Organization",
"site_admin": false
},
"html_url": "https://github.com/git-for-windows/git-for-windows-automation",
"description": "A few GitHub workflows and support code to help with Git for Windows' day-to-day tasks",
"fork": false,
"url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation",
"forks_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/forks",
"keys_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/keys{/key_id}",
"collaborators_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/collaborators{/collaborator}",
"teams_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/teams",
"hooks_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/hooks",
"issue_events_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/issues/events{/number}",
"events_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/events",
"assignees_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/assignees{/user}",
"branches_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/branches{/branch}",
"tags_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/tags",
"blobs_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/blobs{/sha}",
"git_tags_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/tags{/sha}",
"git_refs_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/refs{/sha}",
"trees_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/trees{/sha}",
"statuses_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/statuses/{sha}",
"languages_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/languages",
"stargazers_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/stargazers",
"contributors_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/contributors",
"subscribers_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/subscribers",
"subscription_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/subscription",
"commits_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/commits{/sha}",
"git_commits_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/commits{/sha}",
"comments_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/comments{/number}",
"issue_comment_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/issues/comments{/number}",
"contents_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/contents/{+path}",
"compare_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/compare/{base}...{head}",
"merges_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/merges",
"archive_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/{archive_format}{/ref}",
"downloads_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/downloads",
"issues_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/issues{/number}",
"pulls_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/pulls{/number}",
"milestones_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/milestones{/number}",
"notifications_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/notifications{?since,all,participating}",
"labels_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/labels{/name}",
"releases_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/releases{/id}",
"deployments_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/deployments",
"created_at": "2022-11-28T12:30:12Z",
"updated_at": "2023-11-04T01:08:40Z",
"pushed_at": "2023-11-20T18:58:58Z",
"git_url": "git://github.com/git-for-windows/git-for-windows-automation.git",
"ssh_url": "[email protected]:git-for-windows/git-for-windows-automation.git",
"clone_url": "https://github.com/git-for-windows/git-for-windows-automation.git",
"svn_url": "https://github.com/git-for-windows/git-for-windows-automation",
"homepage": null,
"size": 248,
"stargazers_count": 7,
"watchers_count": 7,
"language": "JavaScript",
"has_issues": true,
"has_projects": false,
"has_downloads": true,
"has_wiki": true,
"has_pages": false,
"has_discussions": true,
"forks_count": 9,
"mirror_url": null,
"archived": false,
"disabled": false,
"open_issues_count": 8,
"license": null,
"allow_forking": true,
"is_template": false,
"web_commit_signoff_required": false,
"topics": [],
"visibility": "public",
"forks": 9,
"open_issues": 8,
"watchers": 7,
"default_branch": "main",
"custom_properties": {}
},
"organization": {
"login": "git-for-windows",
"id": 4571183,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjQ1NzExODM=",
"url": "https://api.github.com/orgs/git-for-windows",
"repos_url": "https://api.github.com/orgs/git-for-windows/repos",
"events_url": "https://api.github.com/orgs/git-for-windows/events",
"hooks_url": "https://api.github.com/orgs/git-for-windows/hooks",
"issues_url": "https://api.github.com/orgs/git-for-windows/issues",
"members_url": "https://api.github.com/orgs/git-for-windows/members{/member}",
"public_members_url": "https://api.github.com/orgs/git-for-windows/public_members{/member}",
"avatar_url": "https://avatars.githubusercontent.com/u/4571183?v=4",
"description": ""
},
"sender": {
"login": "dscho",
"id": 127790,
"node_id": "MDQ6VXNlcjEyNzc5MA==",
"avatar_url": "https://avatars.githubusercontent.com/u/127790?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/dscho",
"html_url": "https://github.com/dscho",
"followers_url": "https://api.github.com/users/dscho/followers",
"following_url": "https://api.github.com/users/dscho/following{/other_user}",
"gists_url": "https://api.github.com/users/dscho/gists{/gist_id}",
"starred_url": "https://api.github.com/users/dscho/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/dscho/subscriptions",
"organizations_url": "https://api.github.com/users/dscho/orgs",
"repos_url": "https://api.github.com/users/dscho/repos",
"events_url": "https://api.github.com/users/dscho/events{/privacy}",
"received_events_url": "https://api.github.com/users/dscho/received_events",
"type": "User",
"site_admin": true
},
"installation": {
"id": 31544731,
"node_id": "MDIzOkludGVncmF0aW9uSW5zdGFsbGF0aW9uMzE1NDQ3MzE="
}
}
}I prefer to teach GitForWindowsHelper much like we did with queued jobs that target Windows/ARM64 to react to such a failed run, extract the name of the VM from the logs of that run (yesterday's log mention actions-runner-20231121103205908804162) and then trigger the delete-self-hosted-runner.yml workflow much like we do when a self-hosted runner is done with a job.
There was a problem hiding this comment.
the GitForWindowsHelper received a
check_runwebhook event with theactionset tocompletedandcheck_run.conclusionset tofailure.
An even better event might be the workflow_job event.
The request that was sent
{
"headers": {
"Accept": "*/*",
"Content-Type": "application/json",
"User-Agent": "GitHub-Hookshot/24a4d61",
"X-GitHub-Delivery": "f7e72c90-885e-11ee-8e1c-f9794b37166a",
"X-GitHub-Event": "workflow_job",
"X-GitHub-Hook-ID": "392499201",
"X-GitHub-Hook-Installation-Target-ID": "246505",
"X-GitHub-Hook-Installation-Target-Type": "integration",
"X-Hub-Signature": "sha1=d76b677038028165db3b5a91a0db8fd8e4deb79d",
"X-Hub-Signature-256": "sha256=2a73803cd0638c9c944672d16d2f38ee18abfa95aa272914593cefb77f47441f"
},
"payload": {
"action": "completed",
"workflow_job": {
"id": 18885592052,
"run_id": 6942503689,
"workflow_name": "create-azure-self-hosted-runners",
"head_branch": "main",
"run_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/actions/runs/6942503689",
"run_attempt": 1,
"node_id": "CR_kwDOIhGD8s8AAAAEZatD9A",
"head_sha": "62c7234114b0ff2590455c0757ab196f0b977dc2",
"url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/actions/jobs/18885592052",
"html_url": "https://github.com/git-for-windows/git-for-windows-automation/actions/runs/6942503689/job/18885592052",
"status": "completed",
"conclusion": "failure",
"created_at": "2023-11-21T10:31:53Z",
"started_at": "2023-11-21T10:32:01Z",
"completed_at": "2023-11-21T11:13:14Z",
"name": "create-runner",
"steps": [
{
"name": "Set up job",
"status": "completed",
"conclusion": "success",
"number": 1,
"started_at": "2023-11-21T10:32:01.000Z",
"completed_at": "2023-11-21T10:32:05.000Z"
},
{
"name": "Generate VM name",
"status": "completed",
"conclusion": "success",
"number": 2,
"started_at": "2023-11-21T10:32:05.000Z",
"completed_at": "2023-11-21T10:32:05.000Z"
},
{
"name": "Run actions/checkout@v3",
"status": "completed",
"conclusion": "success",
"number": 3,
"started_at": "2023-11-21T10:32:06.000Z",
"completed_at": "2023-11-21T10:32:06.000Z"
},
{
"name": "Obtain installation token",
"status": "completed",
"conclusion": "success",
"number": 4,
"started_at": "2023-11-21T10:32:07.000Z",
"completed_at": "2023-11-21T10:32:07.000Z"
},
{
"name": "Generate Actions Runner token and registration URL",
"status": "completed",
"conclusion": "success",
"number": 5,
"started_at": "2023-11-21T10:32:07.000Z",
"completed_at": "2023-11-21T10:32:07.000Z"
},
{
"name": "Azure Login",
"status": "completed",
"conclusion": "success",
"number": 6,
"started_at": "2023-11-21T10:32:07.000Z",
"completed_at": "2023-11-21T10:32:24.000Z"
},
{
"name": "Run azure/arm-deploy@v1",
"status": "completed",
"conclusion": "failure",
"number": 7,
"started_at": "2023-11-21T10:32:25.000Z",
"completed_at": "2023-11-21T11:13:12.000Z"
},
{
"name": "Show post-deployment script output",
"status": "completed",
"conclusion": "skipped",
"number": 8,
"started_at": "2023-11-21T11:13:14.000Z",
"completed_at": "2023-11-21T11:13:14.000Z"
},
{
"name": "Deallocate the VM for later use",
"status": "completed",
"conclusion": "skipped",
"number": 9,
"started_at": "2023-11-21T11:13:14.000Z",
"completed_at": "2023-11-21T11:13:14.000Z"
},
{
"name": "Post Run actions/checkout@v3",
"status": "completed",
"conclusion": "success",
"number": 18,
"started_at": "2023-11-21T11:13:14.000Z",
"completed_at": "2023-11-21T11:13:14.000Z"
},
{
"name": "Complete job",
"status": "completed",
"conclusion": "success",
"number": 19,
"started_at": "2023-11-21T11:13:12.000Z",
"completed_at": "2023-11-21T11:13:12.000Z"
}
],
"check_run_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/check-runs/18885592052",
"labels": [
"ubuntu-latest"
],
"runner_id": 2,
"runner_name": "GitHub Actions 2",
"runner_group_id": 2,
"runner_group_name": "GitHub Actions"
},
"repository": {
"id": 571573234,
"node_id": "R_kgDOIhGD8g",
"name": "git-for-windows-automation",
"full_name": "git-for-windows/git-for-windows-automation",
"private": false,
"owner": {
"login": "git-for-windows",
"id": 4571183,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjQ1NzExODM=",
"avatar_url": "https://avatars.githubusercontent.com/u/4571183?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/git-for-windows",
"html_url": "https://github.com/git-for-windows",
"followers_url": "https://api.github.com/users/git-for-windows/followers",
"following_url": "https://api.github.com/users/git-for-windows/following{/other_user}",
"gists_url": "https://api.github.com/users/git-for-windows/gists{/gist_id}",
"starred_url": "https://api.github.com/users/git-for-windows/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/git-for-windows/subscriptions",
"organizations_url": "https://api.github.com/users/git-for-windows/orgs",
"repos_url": "https://api.github.com/users/git-for-windows/repos",
"events_url": "https://api.github.com/users/git-for-windows/events{/privacy}",
"received_events_url": "https://api.github.com/users/git-for-windows/received_events",
"type": "Organization",
"site_admin": false
},
"html_url": "https://github.com/git-for-windows/git-for-windows-automation",
"description": "A few GitHub workflows and support code to help with Git for Windows' day-to-day tasks",
"fork": false,
"url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation",
"forks_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/forks",
"keys_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/keys{/key_id}",
"collaborators_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/collaborators{/collaborator}",
"teams_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/teams",
"hooks_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/hooks",
"issue_events_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/issues/events{/number}",
"events_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/events",
"assignees_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/assignees{/user}",
"branches_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/branches{/branch}",
"tags_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/tags",
"blobs_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/blobs{/sha}",
"git_tags_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/tags{/sha}",
"git_refs_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/refs{/sha}",
"trees_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/trees{/sha}",
"statuses_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/statuses/{sha}",
"languages_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/languages",
"stargazers_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/stargazers",
"contributors_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/contributors",
"subscribers_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/subscribers",
"subscription_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/subscription",
"commits_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/commits{/sha}",
"git_commits_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/commits{/sha}",
"comments_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/comments{/number}",
"issue_comment_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/issues/comments{/number}",
"contents_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/contents/{+path}",
"compare_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/compare/{base}...{head}",
"merges_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/merges",
"archive_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/{archive_format}{/ref}",
"downloads_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/downloads",
"issues_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/issues{/number}",
"pulls_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/pulls{/number}",
"milestones_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/milestones{/number}",
"notifications_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/notifications{?since,all,participating}",
"labels_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/labels{/name}",
"releases_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/releases{/id}",
"deployments_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/deployments",
"created_at": "2022-11-28T12:30:12Z",
"updated_at": "2023-11-04T01:08:40Z",
"pushed_at": "2023-11-20T18:58:58Z",
"git_url": "git://github.com/git-for-windows/git-for-windows-automation.git",
"ssh_url": "[email protected]:git-for-windows/git-for-windows-automation.git",
"clone_url": "https://github.com/git-for-windows/git-for-windows-automation.git",
"svn_url": "https://github.com/git-for-windows/git-for-windows-automation",
"homepage": null,
"size": 248,
"stargazers_count": 7,
"watchers_count": 7,
"language": "JavaScript",
"has_issues": true,
"has_projects": false,
"has_downloads": true,
"has_wiki": true,
"has_pages": false,
"has_discussions": true,
"forks_count": 9,
"mirror_url": null,
"archived": false,
"disabled": false,
"open_issues_count": 8,
"license": null,
"allow_forking": true,
"is_template": false,
"web_commit_signoff_required": false,
"topics": [],
"visibility": "public",
"forks": 9,
"open_issues": 8,
"watchers": 7,
"default_branch": "main",
"custom_properties": {}
},
"organization": {
"login": "git-for-windows",
"id": 4571183,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjQ1NzExODM=",
"url": "https://api.github.com/orgs/git-for-windows",
"repos_url": "https://api.github.com/orgs/git-for-windows/repos",
"events_url": "https://api.github.com/orgs/git-for-windows/events",
"hooks_url": "https://api.github.com/orgs/git-for-windows/hooks",
"issues_url": "https://api.github.com/orgs/git-for-windows/issues",
"members_url": "https://api.github.com/orgs/git-for-windows/members{/member}",
"public_members_url": "https://api.github.com/orgs/git-for-windows/public_members{/member}",
"avatar_url": "https://avatars.githubusercontent.com/u/4571183?v=4",
"description": ""
},
"sender": {
"login": "gitforwindowshelper[bot]",
"id": 115591365,
"node_id": "BOT_kgDOBuPIxQ",
"avatar_url": "https://avatars.githubusercontent.com/in/246505?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/gitforwindowshelper%5Bbot%5D",
"html_url": "https://github.com/apps/gitforwindowshelper",
"followers_url": "https://api.github.com/users/gitforwindowshelper%5Bbot%5D/followers",
"following_url": "https://api.github.com/users/gitforwindowshelper%5Bbot%5D/following{/other_user}",
"gists_url": "https://api.github.com/users/gitforwindowshelper%5Bbot%5D/gists{/gist_id}",
"starred_url": "https://api.github.com/users/gitforwindowshelper%5Bbot%5D/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/gitforwindowshelper%5Bbot%5D/subscriptions",
"organizations_url": "https://api.github.com/users/gitforwindowshelper%5Bbot%5D/orgs",
"repos_url": "https://api.github.com/users/gitforwindowshelper%5Bbot%5D/repos",
"events_url": "https://api.github.com/users/gitforwindowshelper%5Bbot%5D/events{/privacy}",
"received_events_url": "https://api.github.com/users/gitforwindowshelper%5Bbot%5D/received_events",
"type": "Bot",
"site_admin": false
},
"installation": {
"id": 31544731,
"node_id": "MDIzOkludGVncmF0aW9uSW5zdGFsbGF0aW9uMzE1NDQ3MzE="
}
}
}It shows directly in the payload that the azure/arm-deploy step failed, even though it is still missing the information about the name of the VM that needs to be deleted.
dscho
added a commit
to dscho/gfw-helper-github-app
that referenced
this pull request
Nov 22, 2023
In the previous commit, I added a script to delete existing installations of the GitForWindowsHelper GitHub App for unintended repositories, to avoid cluttering the list of webhook deliveries. I ran this script a couple times in the past, but it was always a bit late for that because I had to click literally over 200 times on that button just to get to payloads that were delivered two days prior. A better way is to use the GitHub REST API to let a script search for specific deliveries, say, for a `check_run` that has completed around a given time. I used this script today to great effect to find the `check_run` and `workflow_job` webhook payloads that I mentioned in git-for-windows/git-for-windows-automation#60 It uses a bit of an ad-hoc strategy to avoid having to load sequential pages of deliveries until the desired time window is reached. That strategy could probably be improved, but it seems to work well enough for now. Signed-off-by: Johannes Schindelin <[email protected]>
|
@dennisameling in the interest not to let the perfect be the enemy of the good, how about dropping f428d14 and then merging this here PR? |
…ariable The image SKU was hidden a bit in the azure-arm-template.json. Let's make it more visible by turning it into a parameter that the CI pipeline can provide. Signed-off-by: Dennis Ameling <[email protected]>
Version 23h2 was released on October 31st, 2023. Let's update to it, so that the runners benefit from the latest improvements to Windows 11 ARM64. Signed-off-by: Dennis Ameling <[email protected]>
Works for me! I've been really time-constrained lately, sorry for not responding earlier. |
dennisameling
force-pushed
the
self-hosted-runner-improvements
branch
from
f428d14
to
db516f6
Compare
dennisameling
changed the title
dscho
approved these changes
Apr 14, 2024
dscho
added a commit
to dscho/git-for-windows-automation
that referenced
this pull request
Sep 14, 2024
This resurrects a GitHub workflow I had originally asked to keep out of git-for-windows#60, when I was concerned that we might delete runner VMs while they are still busy running a workflow. However, in the meantime it happened already twice that I had missed a run-away VM that had not been cleaned up for over a week, which did put a worrisome dent into my Azure credits (and I cannot risk running out of those because a lot of stuff depends on it, e.g. GitGitGadget). So let's merge the commit as-is, in preparation for adjusting it a little bit further in accordance with the experience we gained in the meantime. Signed-off-by: Johannes Schindelin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates the self hosted runner image and also makes it configurable through env vars.
Note: the rest of this PR description was for an older version of this PR and therefore collapsed.
See prior PR description
As discussed here, let's add some cleanup logic for Azure VMs that we no longer need.
In this PR, I added logic that goes through the resource group and deletes leftover VMs. That way, it's not strictly linked to failed runs. I think this is a more robust way to ensure leftover VMs are deleted, but very much open to ideas and feedback.
By "failed run", do you mean the run that creates the self hosted runner, or the run that does the actual build of GfW ARM64? Assuming you mean the former, we'll probably have to add some logic here to detect when a run that creates a self hosted runner has failed, then retry, right? Not sure if that's worth the effort tbh, as these jobs have been rock-solid until I tried again yesterday after some time of not doing it. It probably deserves some more investigation to ensure that VMs are created reliably again.