Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Advice user to use --gpus instead of gpu-addon on docker driver #19114

Open
sivanaikk opened this issue Jun 21, 2024 · 5 comments · May be fixed by #19545
Open

Advice user to use --gpus instead of gpu-addon on docker driver #19114

sivanaikk opened this issue Jun 21, 2024 · 5 comments · May be fixed by #19545
Assignees
@RamBITS-AI
Labels
area/gpu GPU related items good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. kind/improvement Categorizes issue or PR as related to improving upon a current feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@sivanaikk
Copy link

sivanaikk commented Jun 21, 2024 •
edited
Loading

What Happened?

I am trying to integrate GPU with Minikube, My host machine is Windows and I am using docker desktop in wsl2 and trying to provision Minikube with the support of GPU.

I was able to integrate GPU with docker,

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

image

  • Minikube start command

minikube start --driver=docker --addons=ingress --addons=nvidia-gpu-device-plugin
image

image

  • Pod failing with the following error

  Warning  FailedScheduling  45s   default-scheduler  0/1 nodes are available: 1 Insufficient nvidia.com/gpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Attach the log file

log.txt

Operating System

Windows

Driver

Docker

step I followed

Medium Link
@medyagh
Copy link
Member

medyagh commented Jun 21, 2024 •
edited
Loading

@sivanaikk that addon is mean for KVM driver and sorry that minikube did not Warn and advice correctly, this is something we could work on for a better user experience

for docker driver please use --gpus flag

I think the right thing is to Rename this flag to kvm-nvidia-gp and then still accept the people using old name (deprciated name) but for new users it wont be confusing

@medyagh medyagh added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. area/gpu GPU related items kind/improvement Categorizes issue or PR as related to improving upon a current feature. labels Jun 21, 2024
@medyagh medyagh changed the title Unable to integrate GPU with Minikube on Windows Advice user to use --gpus instead of gpu-addon on docker driver Jun 21, 2024
@medyagh medyagh added the good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. label Jun 21, 2024
@sivanaikk
Copy link
Author

Thank you @medyagh. It's working now!

minikube start --gpus all --driver=docker --addons=ingress

image

@ilyesAj ilyesAj mentioned this issue Jun 24, 2024
Open
@medyagh
Copy link
Member

medyagh commented Jun 26, 2024 •
edited
Loading

seems like we have 3 addons,
two of them are for KVM and one is for docker, it is important that we figure out if we can merge some of them

| nvidia-device-plugin | minikube | disabled | 3rd party (NVIDIA) |
| nvidia-driver-installer | minikube | disabled | 3rd party (Nvidia) |
| nvidia-gpu-device-plugin | minikube | disabled | 3rd party (Nvidia) |

https://minikube.sigs.k8s.io/docs/tutorials/nvidia/#docker

in our tutorial seems like, for KVM we need:

  • nvidia-driver-installer
  • nvidia-gpu-device-plugin

for "none" we use:

  • nvidia-device-plugin

so the LEAST we we should do is have a description that this is for KVM only
but ideally we could merge "nvidia-gpu-device-plugin" and "nvidia-device-plugin" so they be one

@spowelljr spowelljr mentioned this issue Jul 4, 2024
Merged
@RamBITS-AI
Copy link

/assign

This was referenced Aug 30, 2024
Closed
Open
@RamBITS-AI
Copy link

I'm starting work (PR - #19545) on this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RamBITS-AI RamBITS-AI

Labels
area/gpu GPU related items good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. kind/improvement Categorizes issue or PR as related to improving upon a current feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Merge nvidia-gpu-device-plugin and nvidia-device-plugin. RamBITS-AI/minikube
3 participants
@medyagh @sivanaikk @RamBITS-AI