Intermittent "Operation cancelled by user" error

[CarolMebiom]

Hi, everyone.

My company has an on-prem kubernetes cluster, set up using k3s, in which we are deploying ephemeral self-hosted runners, with GPU enabled. From time to time we find this error: "Operation cancelled by user" without any intervention on our side. As we were aware that the issue could be resources' constraints, we increased the requests and limits for CPU and memory.

We are using kubernetes v1.26 in Ubuntu machines and the gha runner scale set is of version 0.9.3.

This is the workflow file:

name: CUDA Check with PyTorch

on: [push]

jobs:
  cuda-check:
    runs-on:
        group: RUNNERS
        labels: [ RUNNERS-GPU ]
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.8'

      - name: Install PyTorch with CUDA
        run: |
          pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu118
      - name: Check CUDA availability
        run: |
          python -c "import torch; print('CUDA Available:', torch.cuda.is_available())"
          python -c "import torch; exit(0) if torch.cuda.is_available() else exit(1)"
  cuda-check-mlbase:
    runs-on:
        group: RUNNERS
        labels: [ RUNNERS-GPU ]
    container:
        image: IMAGE
        options: --user root
        credentials:
            username: ${{ vars.DOCKERHUB_USERNAME }}
            password: ${{ secrets.DOCKERHUB_TOKEN }}
    steps:
      - name: Check nvidia-smi
        run: |
            nvidia-smi

Here are the logs of the workflow when it does not finish running because of the "Operation cancelled by user" error:

2024-08-21T15:30:18.0141496Z Current runner version: '2.319.1'
2024-08-21T15:30:18.0146854Z Runner name: 'RUNNERS-GPU-hp9m4-runner-thg9p'
2024-08-21T15:30:18.0147417Z Runner group name: 'RUNNERS'
2024-08-21T15:30:18.0147893Z Machine name: 'RUNNERS-GPU-hp9m4-runner-thg9p'
2024-08-21T15:30:18.0160047Z Testing runner upgrade compatibility
2024-08-21T15:30:18.0689369Z ##[group]GITHUB_TOKEN Permissions
2024-08-21T15:30:18.0690612Z Actions: write
2024-08-21T15:30:18.0690822Z Attestations: write
2024-08-21T15:30:18.0691081Z Checks: write
2024-08-21T15:30:18.0691314Z Contents: write
2024-08-21T15:30:18.0691502Z Deployments: write
2024-08-21T15:30:18.0691745Z Discussions: write
2024-08-21T15:30:18.0691938Z Issues: write
2024-08-21T15:30:18.0692108Z Metadata: read
2024-08-21T15:30:18.0692335Z Packages: write
2024-08-21T15:30:18.0692532Z Pages: write
2024-08-21T15:30:18.0692711Z PullRequests: write
2024-08-21T15:30:18.0692919Z RepositoryProjects: write
2024-08-21T15:30:18.0693214Z SecurityEvents: write
2024-08-21T15:30:18.0693491Z Statuses: write
2024-08-21T15:30:18.0693677Z ##[endgroup]
2024-08-21T15:30:18.0695327Z Secret source: Actions
2024-08-21T15:30:18.0695630Z Prepare workflow directory
2024-08-21T15:30:18.1172832Z Prepare all required actions
2024-08-21T15:30:18.1294923Z Getting action download info
2024-08-21T15:30:18.3679782Z Download action repository 'actions/checkout@v3' (SHA:f43a0e5ff2bd294095638e18286ca9a3d1956744)
2024-08-21T15:30:18.8870366Z Download action repository 'actions/setup-python@v4' (SHA:65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236)
2024-08-21T15:30:19.9556078Z Complete job name: cuda-check
2024-08-21T15:30:20.0169615Z ##[group]Run actions/checkout@v3
2024-08-21T15:30:20.0169895Z with:
2024-08-21T15:30:20.0170092Z   repository: REPO
2024-08-21T15:30:20.0170478Z   token: ***
2024-08-21T15:30:20.0170661Z   ssh-strict: true
2024-08-21T15:30:20.0170863Z   persist-credentials: true
2024-08-21T15:30:20.0171079Z   clean: true
2024-08-21T15:30:20.0171271Z   sparse-checkout-cone-mode: true
2024-08-21T15:30:20.0171526Z   fetch-depth: 1
2024-08-21T15:30:20.0171709Z   fetch-tags: false
2024-08-21T15:30:20.0171896Z   lfs: false
2024-08-21T15:30:20.0172066Z   submodules: false
2024-08-21T15:30:20.0172260Z   set-safe-directory: true
2024-08-21T15:30:20.0172475Z ##[endgroup]
2024-08-21T15:30:20.0878254Z Syncing repository: REPO
2024-08-21T15:30:20.0880047Z ##[group]Getting Git version info
2024-08-21T15:30:20.0880861Z Working directory is '/home/runner/_work/MEDICAL-PRODUCTS/MEDICAL-PRODUCTS'
2024-08-21T15:30:20.0881542Z ##[endgroup]
2024-08-21T15:30:20.0882005Z Deleting the contents of '/home/runner/_work/MEDICAL-PRODUCTS/MEDICAL-PRODUCTS'
2024-08-21T15:30:20.0882506Z The repository will be downloaded using the GitHub REST API
2024-08-21T15:30:20.0882991Z To create a local Git repository instead, add Git 2.18 or higher to the PATH
2024-08-21T15:30:20.0883436Z Downloading the archive
2024-08-21T15:30:27.2925923Z Writing archive to disk
2024-08-21T15:30:27.3467912Z Extracting the archive
2024-08-21T15:30:27.3488504Z [command]/usr/bin/tar xz -C /home/runner/_work/MEDICAL-PRODUCTS/MEDICAL-PRODUCTS/e18821ff-0f94-4ffd-a0b7-ee8f87f343cb -f /home/runner/_work/MEDICAL-PRODUCTS/MEDICAL-PRODUCTS/e18821ff-0f94-4ffd-a0b7-ee8f87f343cb.tar.gz
2024-08-21T15:30:28.0777644Z Resolved version 
2024-08-21T15:30:28.1744873Z ##[group]Run actions/setup-python@v4
2024-08-21T15:30:28.1745159Z with:
2024-08-21T15:30:28.1745340Z   python-version: 3.8
2024-08-21T15:30:28.1745569Z   check-latest: false
2024-08-21T15:30:28.1745867Z   token: ***
2024-08-21T15:30:28.1746053Z   update-environment: true
2024-08-21T15:30:28.1746278Z   allow-prereleases: false
2024-08-21T15:30:28.1746484Z ##[endgroup]
2024-08-21T15:30:28.2702962Z ##[group]Installed versions
2024-08-21T15:30:28.2709522Z Version 3.8 was not found in the local cache
2024-08-21T15:30:28.7842283Z Version 3.8 is available for downloading
2024-08-21T15:30:28.7843059Z Download from "https://github.com/actions/python-versions/releases/download/3.8.18-9962543870/python-3.8.18-linux-22.04-x64.tar.gz"
2024-08-21T15:30:29.9643569Z Extract downloaded archive
2024-08-21T15:30:29.9726775Z [command]/usr/bin/tar xz --warning=no-unknown-keyword -C /home/runner/_work/_temp/6b30c515-c2d7-4d11-a57c-18403b2b9027 -f /home/runner/_work/_temp/73fec934-99d3-4d4d-ab10-8c7690c542de
2024-08-21T15:30:30.8954267Z Execute installation script
2024-08-21T15:30:30.9086771Z Check if Python hostedtoolcache folder exist...
2024-08-21T15:30:30.9087128Z Creating Python hostedtoolcache folder...
2024-08-21T15:30:30.9093629Z Create Python 3.8.18 folder
2024-08-21T15:30:30.9099721Z Copy Python binaries to hostedtoolcache folder
2024-08-21T15:30:31.1712141Z Create additional symlinks (Required for the UsePythonVersion Azure Pipelines task and the setup-python GitHub Action)
2024-08-21T15:30:31.1733504Z Upgrading pip...
2024-08-21T15:30:32.4128408Z Looking in links: /tmp/tmpdw_uoj5a
2024-08-21T15:30:32.4130982Z Requirement already satisfied: setuptools in /home/runner/_work/_tool/Python/3.8.18/x64/lib/python3.8/site-packages (56.0.0)
2024-08-21T15:30:32.4134223Z Requirement already satisfied: pip in /home/runner/_work/_tool/Python/3.8.18/x64/lib/python3.8/site-packages (23.0.1)
2024-08-21T15:30:33.2620619Z Collecting pip
2024-08-21T15:30:33.3386776Z Downloading pip-24.2-py3-none-any.whl (1.8 MB)
2024-08-21T15:30:33.4937112Z ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 12.4 MB/s eta 0:00:00
2024-08-21T15:30:33.4940113Z 
2024-08-21T15:30:33.5427872Z Installing collected packages: pip
2024-08-21T15:30:33.5429147Z Attempting uninstall: pip
2024-08-21T15:30:33.5434278Z Found existing installation: pip 23.0.1
2024-08-21T15:30:33.7100234Z Uninstalling pip-23.0.1:
2024-08-21T15:30:33.7162481Z Successfully uninstalled pip-23.0.1
2024-08-21T15:30:34.2652731Z Successfully installed pip-24.2
2024-08-21T15:30:34.3011951Z Create complete file
2024-08-21T15:30:34.3044969Z Successfully set up CPython (3.8.18)
2024-08-21T15:30:34.3045828Z ##[endgroup]
2024-08-21T15:30:34.3151416Z ##[group]Run pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu118
2024-08-21T15:30:34.3152236Z �[36;1mpip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu118�[0m
2024-08-21T15:30:34.3161574Z shell: /usr/bin/bash -e {0}
2024-08-21T15:30:34.3161809Z env:
2024-08-21T15:30:34.3162064Z   pythonLocation: /home/runner/_work/_tool/Python/3.8.18/x64
2024-08-21T15:30:34.3162510Z   PKG_CONFIG_PATH: /home/runner/_work/_tool/Python/3.8.18/x64/lib/pkgconfig
2024-08-21T15:30:34.3162949Z   Python_ROOT_DIR: /home/runner/_work/_tool/Python/3.8.18/x64
2024-08-21T15:30:34.3163388Z   Python2_ROOT_DIR: /home/runner/_work/_tool/Python/3.8.18/x64
2024-08-21T15:30:34.3163792Z   Python3_ROOT_DIR: /home/runner/_work/_tool/Python/3.8.18/x64
2024-08-21T15:30:34.3164191Z   LD_LIBRARY_PATH: /home/runner/_work/_tool/Python/3.8.18/x64/lib
2024-08-21T15:30:34.3164513Z ##[endgroup]
2024-08-21T15:30:34.5882257Z Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cu118
2024-08-21T15:30:35.2813674Z Collecting torch
2024-08-21T15:30:35.2884564Z   Downloading https://download.pytorch.org/whl/cu118/torch-2.4.0%2Bcu118-cp38-cp38-linux_x86_64.whl (857.7 MB)
2024-08-21T15:30:43.3498053Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 857.7/857.7 MB 88.3 MB/s eta 0:00:00
2024-08-21T15:30:45.4747384Z Collecting torchvision
2024-08-21T15:30:45.4877413Z   Downloading https://download.pytorch.org/whl/cu118/torchvision-0.19.0%2Bcu118-cp38-cp38-linux_x86_64.whl (6.3 MB)
2024-08-21T15:30:45.5652507Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 86.7 MB/s eta 0:00:00
2024-08-21T15:30:46.0380449Z Collecting torchaudio
2024-08-21T15:30:46.0483176Z   Downloading https://download.pytorch.org/whl/cu118/torchaudio-2.4.0%2Bcu118-cp38-cp38-linux_x86_64.whl (3.3 MB)
2024-08-21T15:30:46.0973162Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 68.2 MB/s eta 0:00:00
2024-08-21T15:30:46.2969608Z Collecting filelock (from torch)
2024-08-21T15:30:46.3268371Z   Downloading filelock-3.15.4-py3-none-any.whl.metadata (2.9 kB)
2024-08-21T15:30:46.8054377Z Collecting typing-extensions>=4.8.0 (from torch)
2024-08-21T15:30:46.8114375Z   Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
2024-08-21T15:30:47.0452877Z Collecting sympy (from torch)
2024-08-21T15:30:47.0589667Z   Downloading sympy-1.13.2-py3-none-any.whl.metadata (12 kB)
2024-08-21T15:30:47.3031226Z Collecting networkx (from torch)
2024-08-21T15:30:47.3202513Z   Downloading https://download.pytorch.org/whl/networkx-3.2.1-py3-none-any.whl (1.6 MB)
2024-08-21T15:30:47.3682045Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 40.6 MB/s eta 0:00:00
2024-08-21T15:30:47.5690082Z Collecting jinja2 (from torch)
2024-08-21T15:30:47.5836260Z   Downloading jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
2024-08-21T15:30:48.0878957Z Collecting fsspec (from torch)
2024-08-21T15:30:48.0939163Z   Downloading fsspec-2024.6.1-py3-none-any.whl.metadata (11 kB)
2024-08-21T15:30:48.6192760Z Collecting nvidia-cuda-nvrtc-cu11==11.8.89 (from torch)
2024-08-21T15:30:48.6295385Z   Downloading nvidia_cuda_nvrtc_cu11-11.8.89-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
2024-08-21T15:30:49.0880737Z Collecting nvidia-cuda-runtime-cu11==11.8.89 (from torch)
2024-08-21T15:30:49.0978546Z   Downloading nvidia_cuda_runtime_cu11-11.8.89-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
2024-08-21T15:30:49.2817764Z Collecting nvidia-cuda-cupti-cu11==11.8.87 (from torch)
2024-08-21T15:30:49.2888699Z   Downloading nvidia_cuda_cupti_cu11-11.8.87-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)
2024-08-21T15:30:49.7688347Z Collecting nvidia-cudnn-cu11==9.1.0.70 (from torch)
2024-08-21T15:30:49.7758262Z   Downloading https://download.pytorch.org/whl/cu118/nvidia_cudnn_cu11-9.1.0.70-py3-none-manylinux2014_x86_64.whl (663.9 MB)
2024-08-21T15:30:55.9588248Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 663.9/663.9 MB 95.1 MB/s eta 0:00:00
2024-08-21T15:30:57.4902506Z Collecting nvidia-cublas-cu11==11.11.3.6 (from torch)
2024-08-21T15:30:57.4971179Z   Downloading nvidia_cublas_cu11-11.11.3.6-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
2024-08-21T15:30:57.9532026Z Collecting nvidia-cufft-cu11==10.9.0.58 (from torch)
2024-08-21T15:30:57.9634923Z   Downloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
2024-08-21T15:30:58.4272266Z Collecting nvidia-curand-cu11==10.3.0.86 (from torch)
2024-08-21T15:30:58.4342696Z   Downloading nvidia_curand_cu11-10.3.0.86-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
2024-08-21T15:30:58.6256540Z Collecting nvidia-cusolver-cu11==11.4.1.48 (from torch)
2024-08-21T15:30:58.6406762Z   Downloading nvidia_cusolver_cu11-11.4.1.48-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)
2024-08-21T15:30:58.8293134Z Collecting nvidia-cusparse-cu11==11.7.5.86 (from torch)
2024-08-21T15:30:58.8440287Z   Downloading nvidia_cusparse_cu11-11.7.5.86-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
2024-08-21T15:30:59.3115348Z Collecting nvidia-nccl-cu11==2.20.5 (from torch)
2024-08-21T15:30:59.3234457Z   Downloading https://download.pytorch.org/whl/cu118/nvidia_nccl_cu11-2.20.5-py3-none-manylinux2014_x86_64.whl (142.9 MB)
2024-08-21T15:31:00.6663260Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 142.9/142.9 MB 106.8 MB/s eta 0:00:00
2024-08-21T15:31:01.3561154Z Collecting nvidia-nvtx-cu11==11.8.86 (from torch)
2024-08-21T15:31:01.4065756Z   Downloading nvidia_nvtx_cu11-11.8.86-py3-none-manylinux2014_x86_64.whl.metadata (1.7 kB)
2024-08-21T15:31:01.6681019Z Collecting triton==3.0.0 (from torch)
2024-08-21T15:31:01.6818767Z   Downloading https://download.pytorch.org/whl/triton-3.0.0-1-cp38-cp38-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (209.4 MB)
2024-08-21T15:31:03.6427107Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.4/209.4 MB 106.9 MB/s eta 0:00:00
2024-08-21T15:31:04.6778860Z Collecting numpy (from torchvision)
2024-08-21T15:31:04.6857449Z   Downloading numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
2024-08-21T15:31:05.4021614Z Collecting pillow!=8.3.*,>=5.3.0 (from torchvision)
2024-08-21T15:31:05.4104484Z   Downloading pillow-10.4.0-cp38-cp38-manylinux_2_28_x86_64.whl.metadata (9.2 kB)
2024-08-21T15:31:05.6982308Z Collecting MarkupSafe>=2.0 (from jinja2->torch)
2024-08-21T15:31:05.7083855Z   Downloading https://download.pytorch.org/whl/MarkupSafe-2.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26 kB)
2024-08-21T15:31:05.7370481Z INFO: pip is looking at multiple versions of networkx to determine which version is compatible with other requirements. This could take a while.
2024-08-21T15:31:05.7388952Z Collecting networkx (from torch)
2024-08-21T15:31:05.7483485Z   Downloading networkx-3.1-py3-none-any.whl.metadata (5.3 kB)
2024-08-21T15:31:06.2397158Z Collecting mpmath<1.4,>=1.1.0 (from sympy->torch)
2024-08-21T15:31:06.2551562Z   Downloading https://download.pytorch.org/whl/mpmath-1.3.0-py3-none-any.whl (536 kB)
2024-08-21T15:31:06.2955971Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 9.8 MB/s eta 0:00:00
2024-08-21T15:31:06.3282555Z Downloading nvidia_cublas_cu11-11.11.3.6-py3-none-manylinux2014_x86_64.whl (417.9 MB)
2024-08-21T15:31:10.3876117Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 417.9/417.9 MB 96.9 MB/s eta 0:00:00
2024-08-21T15:31:10.3956834Z Downloading nvidia_cuda_cupti_cu11-11.8.87-py3-none-manylinux2014_x86_64.whl (13.1 MB)
2024-08-21T15:31:10.5464665Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.1/13.1 MB 87.0 MB/s eta 0:00:00
2024-08-21T15:31:10.5565048Z Downloading nvidia_cuda_nvrtc_cu11-11.8.89-py3-none-manylinux2014_x86_64.whl (23.2 MB)
2024-08-21T15:31:10.7913675Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.2/23.2 MB 98.4 MB/s eta 0:00:00
2024-08-21T15:31:10.8078903Z Downloading nvidia_cuda_runtime_cu11-11.8.89-py3-none-manylinux2014_x86_64.whl (875 kB)
2024-08-21T15:31:10.8425977Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 875.6/875.6 kB 27.2 MB/s eta 0:00:00
2024-08-21T15:31:10.8566518Z Downloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux2014_x86_64.whl (168.4 MB)
2024-08-21T15:31:12.4681561Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.4/168.4 MB 104.7 MB/s eta 0:00:00
2024-08-21T15:31:12.4789544Z Downloading nvidia_curand_cu11-10.3.0.86-py3-none-manylinux2014_x86_64.whl (58.1 MB)
2024-08-21T15:31:13.0379106Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.1/58.1 MB 104.3 MB/s eta 0:00:00
2024-08-21T15:31:13.0467516Z Downloading nvidia_cusolver_cu11-11.4.1.48-py3-none-manylinux2014_x86_64.whl (128.2 MB)
2024-08-21T15:31:14.2612683Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 128.2/128.2 MB 105.6 MB/s eta 0:00:00
2024-08-21T15:31:14.2711592Z Downloading nvidia_cusparse_cu11-11.7.5.86-py3-none-manylinux2014_x86_64.whl (204.1 MB)
2024-08-21T15:31:16.1962327Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 204.1/204.1 MB 106.0 MB/s eta 0:00:00
2024-08-21T15:31:16.2033471Z Downloading nvidia_nvtx_cu11-11.8.86-py3-none-manylinux2014_x86_64.whl (99 kB)
2024-08-21T15:31:16.2365787Z Downloading pillow-10.4.0-cp38-cp38-manylinux_2_28_x86_64.whl (4.5 MB)
2024-08-21T15:31:16.2895588Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 93.0 MB/s eta 0:00:00
2024-08-21T15:31:16.2957995Z Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)
2024-08-21T15:31:16.3144580Z Downloading filelock-3.15.4-py3-none-any.whl (16 kB)
2024-08-21T15:31:16.3405467Z Downloading fsspec-2024.6.1-py3-none-any.whl (177 kB)
2024-08-21T15:31:16.3711348Z Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
2024-08-21T15:31:16.4099531Z Downloading networkx-3.1-py3-none-any.whl (2.1 MB)
2024-08-21T15:31:16.4568254Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 53.2 MB/s eta 0:00:00
2024-08-21T15:31:16.4680860Z Downloading numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
2024-08-21T15:31:16.6474303Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 97.5 MB/s eta 0:00:00
2024-08-21T15:31:16.6537596Z Downloading sympy-1.13.2-py3-none-any.whl (6.2 MB)
2024-08-21T15:31:16.7249328Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/6.2 MB 87.5 MB/s eta 0:00:00
2024-08-21T15:31:18.6416242Z Installing collected packages: mpmath, typing-extensions, sympy, pillow, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, numpy, networkx, MarkupSafe, fsspec, filelock, triton, nvidia-cusolver-cu11, nvidia-cudnn-cu11, jinja2, torch, torchvision, torchaudio
**2024-08-21T15:31:38.7026257Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.**
2024-08-21T15:31:39.2715373Z ERROR: Operation cancelled by user
2024-08-21T15:31:39.6019904Z ##[error]The operation was canceled.
2024-08-21T15:31:39.6171068Z Uploading runner diagnostic logs
2024-08-21T15:31:39.6293908Z Completed runner diagnostic log upload
2024-08-21T15:31:39.6294196Z Cleaning up orphan processes

For some reason, the runner receives the shutdown signal. However, the issue is intermittent, sometimes the jobs run, other times it does not. I have looked into the resources and it does not seem to be it, I do not get OOM errors and there are no other pods in the same node requesting GPU...

Besides that, sometimes one of the tasks does not even get pods scheduled even though the Github Self-hosted runner listener presents these logs:

2024-08-21T17:09:15Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":2131,\"replicas\":1}}"}
2024-08-21T17:09:15Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "arc-runners-gpu", "name": "RUNNERS-GPU-hp9m4", "replicas": 1}
2024-08-21T17:09:15Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 111}
2024-08-21T17:10:05Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 0, "decision": 1, "min": 0, "max": 2147483647, "currentRunnerCount": 1, "jobsCompleted": 0}
2024-08-21T17:10:05Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":2132,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2024-08-21T17:10:05Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":2132,\"replicas\":1}}"}
2024-08-21T17:10:05Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "arc-runners-gpu", "name": "RUNNERS-GPU-hp9m4", "replicas": 1}
2024-08-21T17:10:05Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 111}
2024-08-21T17:10:55Z	INFO	listener-app.worker.kubernetesworker	Calculated target runner count	{"assigned job": 0, "decision": 1, "min": 0, "max": 2147483647, "currentRunnerCount": 1, "jobsCompleted": 0}
2024-08-21T17:10:55Z	INFO	listener-app.worker.kubernetesworker	Compare	{"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":2133,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2024-08-21T17:10:55Z	INFO	listener-app.worker.kubernetesworker	Preparing EphemeralRunnerSet update	{"json": "{\"spec\":{\"patchID\":2133,\"replicas\":1}}"}
2024-08-21T17:10:55Z	INFO	listener-app.worker.kubernetesworker	Ephemeral runner set scaled.	{"namespace": "arc-runners-gpu", "name": "RUNNERS-GPU-hp9m4", "replicas": 1}
2024-08-21T17:10:55Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 111}

And the controller presents these ones:

2024-08-21T17:11:45Z	INFO	AutoscalingRunnerSet	Find existing ephemeral runner set	{"version": "0.9.3", "autoscalingrunnerset": {"name":"RUNNERS-GPU","namespace":"arc-runners-gpu"}, "name": "RUNNERS-GPU-hp9m4", "specHash": "6896ffffc9"}
2024-08-21T17:12:35Z	INFO	EphemeralRunnerSet	Ephemeral runner counts	{"version": "0.9.3", "ephemeralrunnerset": {"name":"RUNNERS-GPU-hp9m4","namespace":"arc-runners-gpu"}, "pending": 0, "running": 0, "finished": 0, "failed": 1, "deleting": 0}
2024-08-21T17:12:35Z	INFO	EphemeralRunnerSet	Scaling comparison	{"version": "0.9.3", "ephemeralrunnerset": {"name":"RUNNERS-GPU-hp9m4","namespace":"arc-runners-gpu"}, "current": 1, "desired": 1}
2024-08-21T17:12:35Z	INFO	AutoscalingRunnerSet	Find existing ephemeral runner set	{"version": "0.9.3", "autoscalingrunnerset": {"name":"RUNNERS-GPU","namespace":"arc-runners-gpu"}, "name": "RUNNERS-GPU-hp9m4", "specHash": "6896ffffc9"}
2024-08-21T17:13:25Z	INFO	EphemeralRunnerSet	Ephemeral runner counts	{"version": "0.9.3", "ephemeralrunnerset": {"name":"RUNNERS-GPU-hp9m4","namespace":"arc-runners-gpu"}, "pending": 0, "running": 0, "finished": 0, "failed": 1, "deleting": 0}
2024-08-21T17:13:25Z	INFO	EphemeralRunnerSet	Scaling comparison	{"version": "0.9.3", "ephemeralrunnerset": {"name":"RUNNERS-GPU-hp9m4","namespace":"arc-runners-gpu"}, "current": 1, "desired": 1}
2024-08-21T17:13:25Z	INFO	AutoscalingRunnerSet	Find existing ephemeral runner set	{"version": "0.9.3", "autoscalingrunnerset": {"name":"RUNNERS-GPU","namespace":"arc-runners-gpu"}, "name": "RUNNERS-GPU-hp9m4", "specHash": "6896ffffc9"}

I really do not understand what is going on and whatever help you can suggest is extremely appreciated. If there is any other information you need to provide assistance, please let me know that I will gladly it provide it.