Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Taskfile: Download and set up Meteor.js as part of building the package; Refactor tarball download and extraction into reusable task. #363

Merged
merged 25 commits into from
May 2, 2024
Merged

Taskfile: Download and set up Meteor.js as part of building the package; Refactor tarball download and extraction into reusable task. #363

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
1a09b0c
Taskfile: Add Meteor setup task in Taskfile.yml.
junhaoliao Apr 24, 2024
6dd6bd2
Remove Meteor setup steps from GitHub workflow.
junhaoliao Apr 24, 2024
062fc5c
Revert task "webui-node-modules" back to internal in Taskfile.yml.
junhaoliao Apr 24, 2024
531c5c7
Refactor Taskfile and lint-tasks.yml to extract common download and e…
junhaoliao Apr 24, 2024
930ebbe
Merge branch 'refs/heads/main' into task-meteor-setup
junhaoliao Apr 24, 2024
d3fe1c5
Added webui-node-modules.md5 to sources of task `webui` in Taskfile.yml.
junhaoliao Apr 24, 2024
bc7e85f
Remove temp files resulted from `meteor build` before checksum.
junhaoliao Apr 24, 2024
85409c9
Merge branch 'refs/heads/main' into task-meteor-setup
junhaoliao Apr 26, 2024
60be034
Apply suggestions from code review
junhaoliao Apr 30, 2024
1a13036
Mark task `webui-nodejs` as internal.
junhaoliao Apr 30, 2024
c13f267
Reduce sources array into one-line syntax for `download-tar-and-extra…
junhaoliao Apr 30, 2024
5c62bd5
Add "meteor" to task `webui`'s deps.
junhaoliao Apr 30, 2024
822250d
Rename `download-tar-and-extract` -> ``download-and-extract-tar`.
junhaoliao Apr 30, 2024
705b74a
Rename `FILE_TAR_NAME` -> `TAR_NAME`.
junhaoliao Apr 30, 2024
6242a8f
Refactor `FILE_BASE_DIR` -> `EXTRACTED_DIR` = `OUTPUT_DIR` + ``EXTRAC…
junhaoliao Apr 30, 2024
023f7ad
Change mv command syntax from multiline to single-line string.
junhaoliao Apr 30, 2024
fa2c517
Move "meteor" above "validate-checksum" to stay alphabetical.
junhaoliao Apr 30, 2024
6e0e263
Extract first temporary path then move to preseve hidden files.
junhaoliao Apr 30, 2024
ee73432
Added "run: once" attribute to the task `meteor` to prevent it for ru…
junhaoliao Apr 30, 2024
aedc2fe
Alphabetize vars{}.
junhaoliao Apr 30, 2024
c65e238
Apply suggestions from code review
junhaoliao May 1, 2024
f20b158
Added preconditions to task meteor to prevent running it on Linux ARM.
junhaoliao May 1, 2024
52883fe
Apply suggestions from code review
junhaoliao May 1, 2024
dd2f5b9
Remove Node.js and Meteor.js from the building prerequisites in the d…
junhaoliao May 1, 2024
8a4b223
Merge branch 'refs/heads/main' into task-meteor-setup
junhaoliao May 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 0 additions & 10 deletions .github/workflows/clp-lint.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,6 @@ jobs:
with:
python-version: "3.11"

- id: "parse_meteor_version"
run: |-
release_file="./components/webui/.meteor/release"
meteor_version=$(grep --only-matching --perl-regexp "[\d.]+" $release_file)
echo "meteor_version=${meteor_version}" | tee --append $GITHUB_OUTPUT

- uses: "meteorengineer/setup-meteor@v1"
with:
meteor-release: "${{steps.parse_meteor_version.outputs.meteor_version}}"

- name: "Install task"
shell: "bash"
run: "npm install -g @go-task/cli"
Expand Down
98 changes: 69 additions & 29 deletions Taskfile.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,12 @@ vars:
# Paths
G_BUILD_DIR: "{{.ROOT_DIR}}/build"
G_CORE_COMPONENT_BUILD_DIR: "{{.G_BUILD_DIR}}/core"
G_METEOR_BUILD_DIR: "{{.G_BUILD_DIR}}/meteor"
G_PACKAGE_BUILD_DIR: "{{.G_BUILD_DIR}}/clp-package"
G_PACKAGE_VENV_DIR: "{{.G_BUILD_DIR}}/package-venv"
G_WEBUI_BUILD_DIR: "{{.G_BUILD_DIR}}/webui"
G_WEBUI_NODEJS_BUILD_DIR: "{{.G_BUILD_DIR}}/webui-nodejs"
G_WEBUI_NODEJS_BIN_DIR: "{{.G_WEBUI_NODEJS_BUILD_DIR}}/node/bin"
G_WEBUI_NODEJS_BIN_DIR: "{{.G_WEBUI_NODEJS_BUILD_DIR}}/bin"

# Versions
G_PACKAGE_VERSION: "0.0.3-dev"
Expand Down Expand Up @@ -109,6 +110,7 @@ tasks:
DATA_DIR: "{{.OUTPUT_DIR}}"
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"
sources:
- "{{.G_BUILD_DIR}}/meteor.md5"
junhaoliao marked this conversation as resolved.
Show resolved Hide resolved
- "{{.G_BUILD_DIR}}/package-venv.md5"
- "{{.G_BUILD_DIR}}/webui.md5"
- "{{.G_BUILD_DIR}}/webui-nodejs.md5"
Expand Down Expand Up @@ -182,20 +184,24 @@ tasks:
cmds:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@haiqi96 I noticed we have this platform attribute for task webui. Does it mean the task only runs on x86 platforms? Now that we also set up the arch-specific meteor and nodejs, do we still need the restriction here? (My insights are limited as I'm not too sure how the webui is built currently for your workflow

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does meteor build --architecture now support ARM platform IDs?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the oversight. It doesn't seem to be the case.
The aarch64 related handling was added into the install script together with the 2.14 -> 2.15 upgrade so I mistakenly thought it was available for download. https://static.meteor.com/packages-bootstrap/2.15/meteor-bootstrap-os.linux.aarch64.tar.gz just doesn't open.

ARM support is expected to be available in meteor 3.0 though.

- "rm -rf '{{.OUTPUT_DIR}}'"
- "mkdir -p '{{.OUTPUT_DIR}}'"
- "meteor build --directory '{{.OUTPUT_DIR}}'"
- "PATH='{{.G_METEOR_BUILD_DIR}}':$PATH meteor build --directory '{{.OUTPUT_DIR}}'"
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
- >-
rsync -a
"{{.OUTPUT_DIR}}/bundle/"
launcher.js
settings.json
"{{.OUTPUT_DIR}}/"
- "rm -rf '{{.OUTPUT_DIR}}/bundle/'"
# Remove temp files generated by `meteor build` before checksum
- "find node_modules -type f -name '.meteor-portable-2.json' -exec rm {} +"
# This command must be last
- task: "compute-checksum"
vars:
DATA_DIR: "{{.OUTPUT_DIR}}"
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"
sources:
- "{{.G_BUILD_DIR}}/meteor.md5"
- "{{.G_BUILD_DIR}}/webui-node-modules.md5"
- "{{.TASKFILE}}"
- "*"
- ".meteor/*"
Expand All @@ -209,56 +215,93 @@ tasks:
vars:
CHECKSUM_FILE: "{{.G_BUILD_DIR}}/{{.TASK}}.md5"
OUTPUT_DIR: "{{.G_WEBUI_NODEJS_BUILD_DIR}}"
cmds:
- task: "nodejs"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
NODEJS_VERSION: "v14.21.3"
OUTPUT_DIR: "{{.OUTPUT_DIR}}"

core-submodules:
internal: true
dir: "components/core"
vars:
CHECKSUM_FILE: "{{.G_BUILD_DIR}}/{{.TASK}}.md5"
OUTPUT_DIR: "submodules"
deps:
- "init"
- task: "validate-checksum"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
DATA_DIR: "{{.OUTPUT_DIR}}"
cmds:
- task: "nodejs"
vars:
NODEJS_VERSION: "v14.21.3"
OUTPUT_DIR: "{{.OUTPUT_DIR}}"
- "tools/scripts/deps-download/download-all.sh"
# This command must be last
- task: "compute-checksum"
vars:
DATA_DIR: "{{.OUTPUT_DIR}}"
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"
sources:
- "{{.TASKFILE}}"
- ".gitmodules"
- "tools/scripts/deps-download/**/*"
generates: ["{{.CHECKSUM_FILE}}"]

core-submodules:
download-tar-and-extract:
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
internal: true
dir: "components/core"
vars:
CHECKSUM_FILE: "{{.G_BUILD_DIR}}/{{.TASK}}.md5"
OUTPUT_DIR: "submodules"
requires:
vars: ["CHECKSUM_FILE", "FILE_BASE_DIR", "FILE_TAR_NAME", "OUTPUT_DIR", "URL_PREFIX"]
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
label: "Download & Extract {{.FILE_TAR_NAME}}"
junhaoliao marked this conversation as resolved.
Show resolved Hide resolved
deps:
- "init"
- task: "validate-checksum"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
DATA_DIR: "{{.OUTPUT_DIR}}"
cmds:
- "tools/scripts/deps-download/download-all.sh"
- "rm -rf '{{.OUTPUT_DIR}}'"
- "mkdir -p '{{.OUTPUT_DIR}}'"
- >-
curl --fail --location --show-error
"{{.URL_PREFIX}}/{{.FILE_TAR_NAME}}"
-o "{{.OUTPUT_DIR}}/{{.FILE_TAR_NAME}}"
junhaoliao marked this conversation as resolved.
Show resolved Hide resolved
- "tar xf '{{.OUTPUT_DIR}}/{{.FILE_TAR_NAME}}' --directory '{{.OUTPUT_DIR}}'"
junhaoliao marked this conversation as resolved.
Show resolved Hide resolved
- >-
mv "{{.FILE_BASE_DIR}}/"*
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
"{{.OUTPUT_DIR}}"
- "rm -rf '{{.FILE_BASE_DIR}}'"
- "rm -f '{{.OUTPUT_DIR}}/{{.FILE_TAR_NAME}}'"
junhaoliao marked this conversation as resolved.
Show resolved Hide resolved
# This command must be last
- task: "compute-checksum"
vars:
DATA_DIR: "{{.OUTPUT_DIR}}"
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"
sources:
- "{{.TASKFILE}}"
- ".gitmodules"
- "tools/scripts/deps-download/**/*"
- "{{.TASKFILE_DIR}}/Taskfile.yml"
junhaoliao marked this conversation as resolved.
Show resolved Hide resolved
generates: ["{{.CHECKSUM_FILE}}"]

meteor:
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
vars:
CHECKSUM_FILE: "{{.G_BUILD_DIR}}/{{.TASK}}.md5"
METEOR_RELEASE: "2.15"
METEOR_ARCH: "{{ if eq ARCH \"arm64\" }}aarch64{{ else }}x86_64{{ end }}"
METEOR_PLATFORM: "{{ if eq OS \"darwin\" }}osx{{ else }}linux{{ end }}"
OUTPUT_DIR: "{{.G_METEOR_BUILD_DIR}}"
METEOR_FILE_BASE_DIR: "{{.OUTPUT_DIR}}/.meteor"
junhaoliao marked this conversation as resolved.
Show resolved Hide resolved
cmds:
- task: "download-tar-and-extract"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
FILE_TAR_NAME: "meteor-bootstrap-os.{{.METEOR_PLATFORM}}.{{.METEOR_ARCH}}.tar.gz"
FILE_BASE_DIR: "{{.OUTPUT_DIR}}/.meteor"
junhaoliao marked this conversation as resolved.
Show resolved Hide resolved
URL_PREFIX: "https://static.meteor.com/packages-bootstrap/{{.METEOR_RELEASE}}"
OUTPUT_DIR: "{{.OUTPUT_DIR}}"
junhaoliao marked this conversation as resolved.
Show resolved Hide resolved

nodejs:
internal: true
deps: ["init"]
requires:
vars: ["NODEJS_VERSION", "OUTPUT_DIR"]
vars: ["CHECKSUM_FILE", "NODEJS_VERSION", "OUTPUT_DIR"]
vars:
NODEJS_ARCH: "{{ if eq ARCH \"arm64\" }}arm64{{ else }}x64{{ end }}"
NODEJS_VERSION_BASE_URL: "https://nodejs.org/dist/{{.NODEJS_VERSION}}/"
Expand All @@ -271,19 +314,14 @@ tasks:
--max-count 1
"node-v\\d+\\.\\d+\\.\\d+-linux-{{.NODEJS_ARCH}}"
| head --lines 1
NODEJS_FILE_TAR_NAME: "{{.NODEJS_FILE_BASE_NAME}}.tar.xz"
cmds:
- "rm -rf '{{.OUTPUT_DIR}}/node'"
- "mkdir -p '{{.OUTPUT_DIR}}'"
- >-
curl -fsSL
"{{.NODEJS_VERSION_BASE_URL}}{{.NODEJS_FILE_TAR_NAME}}"
-o "{{.OUTPUT_DIR}}/{{.NODEJS_FILE_TAR_NAME}}"
- "tar xf '{{.OUTPUT_DIR}}/{{.NODEJS_FILE_TAR_NAME}}' --directory '{{.OUTPUT_DIR}}'"
- >-
mv "{{.OUTPUT_DIR}}/{{.NODEJS_FILE_BASE_NAME}}"
"{{.OUTPUT_DIR}}/node"
- "rm -f '{{.OUTPUT_DIR}}/{{.NODEJS_FILE_TAR_NAME}}'"
- task: "download-tar-and-extract"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
FILE_TAR_NAME: "{{.NODEJS_FILE_BASE_NAME}}.tar.xz"
FILE_BASE_DIR: "{{.OUTPUT_DIR}}/{{.NODEJS_FILE_BASE_NAME}}"
URL_PREFIX: "{{.NODEJS_VERSION_BASE_URL}}"
OUTPUT_DIR: "{{.OUTPUT_DIR}}"

package-venv:
internal: true
Expand Down Expand Up @@ -358,15 +396,17 @@ tasks:
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
DATA_DIR: "{{.OUTPUT_DIR}}"
- "meteor"
kirkrodrigues marked this conversation as resolved.
Show resolved Hide resolved
cmds:
- "rm -rf '{{.OUTPUT_DIR}}'"
- "meteor npm install --production"
- "PATH='{{.G_METEOR_BUILD_DIR}}':$PATH meteor npm install --production"
# This command must be last
- task: "compute-checksum"
vars:
DATA_DIR: "{{.OUTPUT_DIR}}"
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"
sources:
- "{{.G_BUILD_DIR}}/meteor.md5"
- "{{.TASKFILE}}"
- ".meteor/packages"
- "package.json"
Expand Down
18 changes: 2 additions & 16 deletions lint-tasks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: "3"

vars:
G_LINTER_NODEJS_BUILD_DIR: "{{.G_BUILD_DIR}}/linter-nodejs"
G_LINTER_NODEJS_BIN_DIR: "{{.G_LINTER_NODEJS_BUILD_DIR}}/node/bin"
G_LINTER_NODEJS_BIN_DIR: "{{.G_LINTER_NODEJS_BUILD_DIR}}/bin"
G_LINT_VENV_DIR: "{{.G_BUILD_DIR}}/lint-venv"

tasks:
Expand Down Expand Up @@ -145,26 +145,12 @@ tasks:
vars:
CHECKSUM_FILE: "{{.G_BUILD_DIR}}/{{.TASK | replace \":\" \"#\"}}.md5"
OUTPUT_DIR: "{{.G_LINTER_NODEJS_BUILD_DIR}}"
deps:
- ":init"
- task: ":validate-checksum"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
DATA_DIR: "{{.OUTPUT_DIR}}"
cmds:
- task: ":nodejs"
vars:
CHECKSUM_FILE: "{{.CHECKSUM_FILE}}"
NODEJS_VERSION: "latest"
OUTPUT_DIR: "{{.OUTPUT_DIR}}"
# This command must be last
- task: ":compute-checksum"
vars:
DATA_DIR: "{{.OUTPUT_DIR}}"
OUTPUT_FILE: "{{.CHECKSUM_FILE}}"
sources:
- "{{.ROOT_DIR}}/Taskfile.yml"
- "{{.TASKFILE}}"
generates: ["{{.CHECKSUM_FILE}}"]

linter-node-modules:
internal: true
Expand Down
Loading