/cc @machadovilaca

Although I don't know in detail how we are handling GPUs in KubeVirt, I think it's clear that we passthrough the device name exactly, and it seems this is related to the NVIDIA metric collector. If you set up your VM requesting a GPU

spec:
  domain:
    devices:
      ...
      gpus:
      - deviceName: nvidia.com/SOMETHING
        name: gpu1

the resulting virt-launcher pod will contain a direct match of that label as expected

resources:
  ...
  requests:
    ...
    nvidia.com/SOMETHING: "1"

but the NVIDIA exporter, as you mentioned, only cares about resources named exactly nvidia.com/gpu or prefixed with nvidia.com/mig-. In the "GPU bound directly to a pod" you mentioned it's working, what is the GPU device name you are using?

example:
https://github.com/machadovilaca/kubevirt/blob/test-nvidia-dcgm-exporter/tests/monitoring/gpu.go

Thanks so much for focusing on this! Here's the yaml for kubevirt binding to the gpu.

  resources:
     requests:
       ...
       nvidia.com/gpu: 1 
     limits:
       ...
       nvidia.com/gpu: 1 

I don't need to specify which GPU card to use when binding the GPU directly to the pod. I only need to focus on the number of GPUs bound to the pod, which will be handled by the GPU Opreator, so I just need to configure it as follows

If I set the deviceName to nvidia.com/gpu in kubevirt's passthrough mode I could theoretically achieve monitoring, but that doesn't seem very reasonable. And there's no way for the user to realize that this is needed to make the GPUs bound to the kubevirt luancher pod available for monitoring.

Instead, I would prefer that the kubevirt luancher pod set the podResource Name to nvidia.com/gpu regardless of the name of the GPU card if it is bound to a Navid GPU.

Of course, a better solution would be to push DCMG Export to change the policy of forcing matching nvidia.com/gpu to instead match the nvidia.com/gpu- prefix as the policy.

Why is it not reasonable? Per my understanding, if you request a specific GPU for the pod, you also won't have NVIDIA monitoring there.

If you ask KubeVirt to request a specific nvidia.com/SOMETHING GPU to the VM, I don't think it makes sense for KubeVirt to decide "we don't care which GPU, let's just request nvidia.com/gpu: 1".

I think you're right.

it's not practical to try to fix this in kubevirt. I have some GPU cards mounted in my cluster and from kubectl describe node I can get the following information.

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                       Requests           Limits
  --------                       --------           ------
  ...
  nvidia.com/GP104GL_TESLA_P4    2                  2
  nvidia.com/GRID_P4-1Q          0                  0
  nvidia.com/GRID_P4-4Q          0                  0

if request nvidia.com/gpu: "1" , This is not recognized by k8s and the vm can not been created success.

This appears to be because the DCGM Exporter strictly follows the k8s specification for determining GPU resource. refer k8s device plugin,

The ResourceName it wants to advertise. Here ResourceName needs to follow the extended resource naming scheme as vendor-domain/resourcetype. (For example, an NVIDIA GPU is advertised as nvidia.com/gpu.)

This issue is better suited for upstream discussion, and I'll probably cite this issue as a practical example of what I'm looking for.

Thanks for the help provided!

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

@kubevirt-bot: Closing this issue.