Integration for Buildkite #4294

udsamani · 2024-07-31T01:06:56Z

This PR aims at the following

Create a buildkite-hosted-agent for bacalhau
Create a bacalhau-golang pipeline
Add buildkite scripts for different components

coderabbitai · 2024-07-31T01:07:04Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

This reverts commit fa0c02f.

wdbaruni · 2024-07-31T07:37:28Z

buildkite/scripts/test.sh

+make build-webui
+make test-and-report


do we need to build the webui with each test?

wdbaruni · 2024-07-31T07:39:44Z

docker/buildkite-hosted-agent-bacalhau/Dockerfile

@@ -0,0 +1,43 @@
+FROM buildkite/hosted-agent-base


Nice. Much cleaner that circleci setup. Can we put everything related to buildkite under /buidlkite directory, including this Dockerfile?

wdbaruni · 2024-07-31T07:42:47Z

buildkite/pipelines/bacalhau-golang.yaml

+cache:
+  paths:
+    - "/golangci-lint-cache"
+    - "/vendor-cache"
+    - "/node-modules-cache"
+  size: 20g
+  name: "vendor-module-cache"


how is this cache used? We also want to make sure we build once, and then use the builds in later phases, such as running unit tests, integration tests, and releasing them if this build includes a release

also do we pull the code once, or every step in the pipeline pulls the code again?

Every step in pipeline unfortunately pulls code :( But we use git mirroring which fastens up.

If you want to do just once. You cant take advantage of parallel builds.

Circle CI was the same too.

wdbaruni · 2024-07-31T07:45:20Z

buildkite/pipelines/bacalhau-golang.yaml

+  - group: ":package: Build Tarball"
+    key: "build-tarball"
+    steps:
+      - label: ":golang: Build linux amd64"
+        command: "./buildkite/scripts/build_tarball.sh linux amd64"
+        agents:
+          queue: "buildkite-hosted-linux-small"
+
+      - label: ":golang: Build linux arm64"
+        command: "./buildkite/scripts/build_tarball.sh linux arm64"
+        agents:
+          queue: "buildkite-hosted-linux-small"


how are steps in the same group executed? Do each require spinning up their own container, which means slower start and cost, or are they run within the same agent/container? I am double checking because we define the agents in each step, which implies they run on separate environments

They are on different agents. Spinning up the agent is pretty fast. It stays in queue for max 5-6s. Group step only helps seeing things visually !

wdbaruni · 2024-07-31T07:47:51Z

buildkite/pipelines/bacalhau-golang.yaml

+  - label: ":testengine: Unit Test"
+    command: "./buildkite/scripts/test.sh unit"
+    key: "unit-test"
+    agents:
+      queue: "buildkite-hosted-linux-large"
+
+  - label: ":testengine: Integration Test"
+    command: "./buildkite/scripts/test.sh integration"
+    key: "integration-test"
+    agents:
+      queue: "buildkite-hosted-linux-large"


eventually we want to run on different environments, including windows, linux arm, linux amd, and darwin.

Also, separating unit and integration tests into separate steps was done before because of consistent failures in integration tests and to be able to retry only the failed steps in circleci instead of the whole tests again. We can continue to do the same here as well if it makes sense, but we should re-evaluate if this is adding unnecessary cost

wdbaruni · 2024-07-31T08:00:43Z

buildkite/scripts/build_tarball.sh

+#!/bin/bash
+
+GOOS=$1 GOARCH=$2 make build-bacalhau-tgz


I looked at the build logs https://buildkite.com/expanso/bacalhau-golang/builds/291#01910786-63c9-4a7e-b38d-d12c63b50496

some observations:

It seems we are building the webui everytime. This is an issue with make target setup

I see some failures at the end related to signing the builds because of missing /tmp/private.pem. Check

bacalhau/.circleci/config.yml

Line 317 in e71b30b

name: Build tarball

on how we were doing things with circleci

Yess ! I want to have at least one green build with buildkite before I merge this. I am debugging those.

This reverts commit 4e5d92b.

wdbaruni · 2024-08-07T12:13:26Z

pkg/executor/docker/executor_test.go

@@ -381,6 +384,7 @@ func (s *ExecutorTestSuite) TestDockerNetworkingFiltersHTTP() {
 }

 func (s *ExecutorTestSuite) TestDockerNetworkingFiltersHTTPS() {
+	s.T().Skip("Skipping the test until buildkite is fixed.")


nit: will be nice to only skip in buildkite and let them run when testing locally. Env variables can help with that

wdbaruni · 2024-08-07T12:14:19Z

pkg/executor/docker/gateway/gateway.sh

-# Apply rate limits to the outbound connections. We just do this for all
-# interfaces rather than working out which is our Internet connection.
-while IFS= read -r IFACE; do
-    tc qdisc add dev "${IFACE}" root tbf rate 10mbit burst 32kbit latency 10sec
-done < <(ip --json address show | jq -rc '.[] | .ifname')
-


do we need to remove these even if we are skipping the tests? If yes, then please add an issue to make sure we fix this way before 1.5

Done. We don't need to remove this. So reverted it.

udsamani added 11 commits July 30, 2024 21:55

add hosted bacalhau buildkite image

92c50f6

update image

c66be0f

add linting script

cc17bdc

add build script

1ca3f66

update build script

eb1171d

update build script

bbae6b4

update build script

7bdc7ff

update build script

d2538fa

add building for webui

a0c54ab

add unit testing for linux

dd517ba

webui and unit test should be in 1 script

4893769

udsamani added the build-ready label Jul 31, 2024

remove circle-ci triggers

fa0c02f

udsamani self-assigned this Jul 31, 2024

udsamani added build/go A label to indicate buildkite to trigger golang builds for this PR and removed build-ready labels Jul 31, 2024

Revert "remove circle-ci triggers"

9308011

This reverts commit fa0c02f.

udsamani requested a review from wdbaruni July 31, 2024 06:39

add bacalha-golang pipeline

671c615

wdbaruni reviewed Jul 31, 2024

View reviewed changes

udsamani added 8 commits August 1, 2024 09:58

try qdisc fix

c4210fd

web ui shouldnt' be building always

4e5d92b

Revert "web ui shouldnt' be building always"

a1eb568

This reverts commit 4e5d92b.

skip failing qdisc tests

50a5013

keys are fixed

70a59b7

try fix for failing unit tests

e30c749

try fix for failing unit tests

ffafcb5

try fix for failing unit tests

e0b59fa

udsamani added 5 commits August 2, 2024 12:02

try fix for failing unit tests

fd7304e

try fix for failing unit tests

508f733

uploading test results

1c12ab6

uploading test results

ea81e65

uploading test results

99126ff

udsamani requested a review from wdbaruni August 2, 2024 12:19

python pipeline

03ac814

udsamani added the build/python label Aug 3, 2024

udsamani added 3 commits August 3, 2024 20:20

add python testing

a6f9b1c

parallelization of tests try

a707e1a

parallelization of tests try 2

bfaad8e

wdbaruni approved these changes Aug 7, 2024

View reviewed changes

udsamani added 7 commits August 8, 2024 00:43

read s3 credentials from buildkite agent

b5c6e8d

read s3 credentials from buildkite agent

952df7a

ipfs daemon run

0303652

ipfs init before daemon run

4bfcccf

revert gateway.sh and fix flaky test, as we should always select port

f318230

fix assert condition after free port

8492243

Merge remote-tracking branch 'origin' into buildkite

9a15266

udsamani merged commit a3dcc8f into main Aug 8, 2024
9 of 11 checks passed

udsamani deleted the buildkite branch August 8, 2024 00:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integration for Buildkite #4294

Integration for Buildkite #4294

udsamani commented Jul 31, 2024 •

edited

Loading

coderabbitai bot commented Jul 31, 2024 •

edited

Loading

Review skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

wdbaruni Jul 31, 2024

wdbaruni Jul 31, 2024

wdbaruni Jul 31, 2024

wdbaruni Jul 31, 2024

udsamani Jul 31, 2024

wdbaruni Jul 31, 2024

udsamani Jul 31, 2024

wdbaruni Jul 31, 2024

wdbaruni Jul 31, 2024

udsamani Jul 31, 2024

wdbaruni Aug 7, 2024

wdbaruni Aug 7, 2024

udsamani Aug 8, 2024

Integration for Buildkite #4294

Integration for Buildkite #4294

Conversation

udsamani commented Jul 31, 2024 • edited Loading

coderabbitai bot commented Jul 31, 2024 • edited Loading

Review skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

udsamani commented Jul 31, 2024 •

edited

Loading

coderabbitai bot commented Jul 31, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)