GPU runners and dind

[awoimbee]

Hi,
I saw 0 information about GPU runners, so I just wanted to say that is works great !
A bit of tinkering is needed to get dind with GPU working tho. I built a custom dind container to allow safe gpu passthrough (check it out).
You will need to set the dind container to use this image in the deployment config of the controller (current helm chart: image.dindSidecarRepositoryAndTag).

Currently, docker:dind is 230MB and nvidia-dind:latest is 653MB, the nvidia-dind container will work just fine on non gpu nodes too. ;)

[xuzhao9]

Thanks, I am wondering how to specify different compose options for dind? For example, I would like to run the dind image nvidia/cuda:11.7.1-cudnn8-devel-ubuntu20.04 with the following arguments:

docker run \
  -it \
  --volume /home/kubernetes/bin/nvidia/lib64:/usr/local/nvidia/lib64 \
  --volume /home/kubernetes/bin/nvidia/bin:/usr/local/nvidia/bin \
  --device /dev/nvidia0:/dev/nvidia0 \
  --device /dev/nvidia-uvm:/dev/nvidia-uvm \
  --device /dev/nvidiactl:/dev/nvidiactl \
  --cap-add=SYS_ADMIN \
  nvidia/cuda:11.7.1-cudnn8-devel-ubuntu20.04 entrypoint.sh

This is because we are using Google Cloud's container-optimized-system as the instance operating system, which does not support nvidia docker daemon.