From c1ca7d6d1f4da8f4b3bafa7b641b857748640aa3 Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Thu, 29 Aug 2024 08:15:40 +0000 Subject: [PATCH 01/19] NVIDIA: SAUCE: vfio/nvgrace-egm: Introduce module to manage EGM BugLink: https://bugs.launchpad.net/bugs/2119656 The Extended GPU Memory (EGM) feature enables the GPU access to the system memory across sockets and nodes. In this mode, the physical memory can be allocated for GPU usage from anywhere in a multi-node system. The feature is being extended to virtualization. EGM when enabled in the virtualization stack, the host memory is partitioned into 2: One partition for the Host OS usage, and a second EGM region. The EGM region essentially becomes the system memory of the VM. The following figure shows the memory map in the virtualization environment. |---- Sysmem ----| |--- GPU mem ---| VM Memory Map | | | | | | | | |------ EGM -----|--Host Mem----| |--- GPU mem ---| Host Memory Map The EGM region is not available to the host memory for its usage as it is not added to the kernel. Its base HPA and the length is communicated through the DSDT entries. A linear mapping between the VM IPA and system HPA is a requirement for EGM support. The EGM region is thus assigned to a VM by mapping the QEMU VMA to a linearly increasing HPA of the EGM region using remap_pfn_range(). Introduce a new nvgrace-egm helper module to nvgrace-gpu to manage the EGM/VM region for the VM. nvgrace-egm module handles the following: 1. Fetch the EGM memory properties (base HPA, length, proximity domain). 2. Create a char device that can be used as memory-backend-file by Qemu for the VM and implement file operations. The char device is /dev/egmX, where X is the PXM node ID of the EGM being mapped fetched in 1. 3. Zero the EGM memory on first device open(). 4. Map the QEMU VMA to the EGM region using remap_pfn_range. 5. Cleaning up state and destroying the chardev on device unbind. Signed-off-by: Ankit Agrawal Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit 892ac2417c614969ff215ad75c0249af6073ffb9 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 3a1b8196060afeaec7b37a1300706d59642e8212 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (cherry picked from commit 8807f4b90409 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (backported from commit fa304984adc4ac07d2fd10d68c1cea99e2b7c12f noble:linux-nvidia-6.17) [jacobmartin: adjust patch context to align with upstream commit e5f19b619fa0 ("vfio/nvgrace-gpu: register device memory for poison handling"), as opposed to the original SAUCE version of the same patch.] Signed-off-by: Jacob Martin (cherry picked from commit 5a1e11bc5c36712f9cc40a1509c70f5d10cc374b https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/Kconfig | 11 ++ drivers/vfio/pci/nvgrace-gpu/Makefile | 3 + drivers/vfio/pci/nvgrace-gpu/egm.c | 235 ++++++++++++++++++++++++++ drivers/vfio/pci/nvgrace-gpu/egm.h | 12 ++ drivers/vfio/pci/nvgrace-gpu/main.c | 31 +++- 5 files changed, 291 insertions(+), 1 deletion(-) create mode 100644 drivers/vfio/pci/nvgrace-gpu/egm.c create mode 100644 drivers/vfio/pci/nvgrace-gpu/egm.h diff --git a/drivers/vfio/pci/nvgrace-gpu/Kconfig b/drivers/vfio/pci/nvgrace-gpu/Kconfig index a7f624b37e410..d5773bbd22f5e 100644 --- a/drivers/vfio/pci/nvgrace-gpu/Kconfig +++ b/drivers/vfio/pci/nvgrace-gpu/Kconfig @@ -1,8 +1,19 @@ # SPDX-License-Identifier: GPL-2.0-only +config NVGRACE_EGM + tristate "EGM driver for NVIDIA Grace Hopper and Blackwell Superchip" + depends on ARM64 || (COMPILE_TEST && 64BIT) + help + Extended GPU Memory (EGM) support for the GPU in the NVIDIA Grace + based chips required to avail the CPU memory as additional + cross-node/cross-socket memory for GPU using KVM/qemu. + + If you don't know what to do here, say N. + config NVGRACE_GPU_VFIO_PCI tristate "VFIO support for the GPU in the NVIDIA Grace Hopper Superchip" depends on ARM64 || (COMPILE_TEST && 64BIT) select VFIO_PCI_CORE + select NVGRACE_EGM help VFIO support for the GPU in the NVIDIA Grace Hopper Superchip is required to assign the GPU device to userspace using KVM/qemu/etc. diff --git a/drivers/vfio/pci/nvgrace-gpu/Makefile b/drivers/vfio/pci/nvgrace-gpu/Makefile index 3ca8c187897a9..c99b04a94e770 100644 --- a/drivers/vfio/pci/nvgrace-gpu/Makefile +++ b/drivers/vfio/pci/nvgrace-gpu/Makefile @@ -1,3 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu-vfio-pci.o nvgrace-gpu-vfio-pci-y := main.o + +obj-$(CONFIG_NVGRACE_EGM) += nvgrace-egm.o +nvgrace-egm-y := egm.o diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c new file mode 100644 index 0000000000000..f3c22a9dfecb9 --- /dev/null +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#include +#include "egm.h" + +#define MAX_EGM_NODES 256 + +struct egm_region { + struct list_head list; + int egmpxm; + atomic_t open_count; + phys_addr_t egmphys; + size_t egmlength; + struct device device; + struct cdev cdev; +}; + +static dev_t dev; +static struct class *class; +static struct list_head egm_list; + +static int nvgrace_egm_open(struct inode *inode, struct file *file) +{ + void *memaddr; + struct egm_region *region = container_of(inode->i_cdev, + struct egm_region, cdev); + + if (!region) + return -EINVAL; + + if (atomic_inc_return(®ion->open_count) > 1) + return 0; + + memaddr = memremap(region->egmphys, region->egmlength, MEMREMAP_WB); + if (!memaddr) { + atomic_dec(®ion->open_count); + return -EINVAL; + } + + memset((u8 *)memaddr, 0, region->egmlength); + memunmap(memaddr); + file->private_data = region; + + return 0; +} + +static int nvgrace_egm_release(struct inode *inode, struct file *file) +{ + struct egm_region *region = container_of(inode->i_cdev, + struct egm_region, cdev); + + if (!region) + return -EINVAL; + + if (atomic_dec_and_test(®ion->open_count)) + file->private_data = NULL; + + return 0; +} + +static int nvgrace_egm_mmap(struct file *file, struct vm_area_struct *vma) +{ + int ret = 0; + struct egm_region *region = file->private_data; + + if (!region) + return -EINVAL; + + ret = remap_pfn_range(vma, vma->vm_start, + PHYS_PFN(region->egmphys), + (vma->vm_end - vma->vm_start), + vma->vm_page_prot); + return ret; +} + +static const struct file_operations file_ops = { + .owner = THIS_MODULE, + .open = nvgrace_egm_open, + .release = nvgrace_egm_release, + .mmap = nvgrace_egm_mmap, +}; + +static int setup_egm_chardev(struct egm_region *region) +{ + int ret = 0; + + device_initialize(®ion->device); + + /* + * Use the proximity domain number as the device minor + * number. So the EGM corresponding to node X would be + * /dev/egmX. + */ + region->device.devt = MKDEV(MAJOR(dev), region->egmpxm); + region->device.class = class; + cdev_init(®ion->cdev, &file_ops); + region->cdev.owner = THIS_MODULE; + + ret = dev_set_name(®ion->device, "egm%d", region->egmpxm); + if (ret) + return ret; + + ret = cdev_device_add(®ion->cdev, ®ion->device); + + return ret; +} + +static int +nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, + u64 *pegmlength, u64 *pegmpxm) +{ + int ret; + + /* + * The memory information is present in the system ACPI tables as DSD + * properties nvidia,egm-base-pa and nvidia,egmm-size. + */ + ret = device_property_read_u64(&pdev->dev, "nvidia,egm-size", + pegmlength); + if (ret) + return ret; + + if (*pegmlength > type_max(size_t)) + return -EOVERFLOW; + + ret = device_property_read_u64(&pdev->dev, "nvidia,egm-base-pa", + pegmphys); + if (ret) + return ret; + + if (*pegmphys > type_max(phys_addr_t)) + return -EOVERFLOW; + + ret = device_property_read_u64(&pdev->dev, "nvidia,egm-pxm", + pegmpxm); + + if (*pegmpxm > type_max(phys_addr_t)) + return -EOVERFLOW; + + return ret; +} + +int register_egm_node(struct pci_dev *pdev) +{ + struct egm_region *region = NULL; + u64 egmphys, egmlength, egmpxm; + int ret; + + ret = nvgrace_gpu_fetch_egm_property(pdev, &egmphys, &egmlength, &egmpxm); + if (ret) + return ret; + + list_for_each_entry(region, &egm_list, list) { + if (region->egmphys == egmphys) + return 0; + } + + region = kvzalloc(sizeof(*region), GFP_KERNEL); + region->egmphys = egmphys; + region->egmlength = egmlength; + region->egmpxm = egmpxm; + + atomic_set(®ion->open_count, 0); + + list_add_tail(®ion->list, &egm_list); + + setup_egm_chardev(region); + + return 0; +} +EXPORT_SYMBOL_GPL(register_egm_node); + +static void destroy_egm_chardev(struct egm_region *region) +{ + cdev_device_del(®ion->cdev, ®ion->device); +} + +void unregister_egm_node(int egm_node) +{ + struct egm_region *region, *temp_region; + + list_for_each_entry_safe(region, temp_region, &egm_list, list) { + if (egm_node == region->egmpxm) { + destroy_egm_chardev(region); + list_del(®ion->list); + } + } +} +EXPORT_SYMBOL_GPL(unregister_egm_node); + +static char *egm_devnode(const struct device *device, umode_t *mode) +{ + if (mode) + *mode = 0600; + + return NULL; +} + +static int __init nvgrace_egm_init(void) +{ + int ret; + + ret = alloc_chrdev_region(&dev, + 0, MAX_EGM_NODES, "egm"); + if (ret < 0) + return ret; + + class = class_create("egm"); + if (IS_ERR(class)) { + unregister_chrdev_region(dev, MAX_EGM_NODES); + return PTR_ERR(class); + } + + class->devnode = egm_devnode; + + INIT_LIST_HEAD(&egm_list); + + return 0; +} + +static void __exit nvgrace_egm_cleanup(void) +{ + class_destroy(class); + unregister_chrdev_region(dev, MAX_EGM_NODES); +} + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ankit Agrawal "); +MODULE_DESCRIPTION("NVGRACE EGM - Helper module of NVGRACE GPU to support Extended GPU Memory"); + +module_init(nvgrace_egm_init); +module_exit(nvgrace_egm_cleanup); diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.h b/drivers/vfio/pci/nvgrace-gpu/egm.h new file mode 100644 index 0000000000000..28cc59e04a0b0 --- /dev/null +++ b/drivers/vfio/pci/nvgrace-gpu/egm.h @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#ifndef NVGRACE_EGM_H +#define NVGRACE_EGM_H + +int register_egm_node(struct pci_dev *pdev); +void unregister_egm_node(int egm_node); + +#endif /* NVGRACE_EGM_H */ diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index 4232ce33a2591..0ef849da0e19c 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -10,6 +10,7 @@ #include #include #include +#include "egm.h" /* * The device memory usable to the workloads running in the VM is cached @@ -64,8 +65,11 @@ struct nvgrace_gpu_pci_core_device { bool has_mig_hw_bug; /* GPU has just been reset */ bool reset_done; + int egm_node; }; +static bool egm_enabled; + static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device *core_vdev) { struct nvgrace_gpu_pci_core_device *nvdev = @@ -1041,6 +1045,13 @@ nvgrace_gpu_fetch_memory_property(struct pci_dev *pdev, return ret; } +static int +nvgrace_gpu_has_egm_property(struct pci_dev *pdev, u64 *pegmpxm) +{ + return device_property_read_u64(&pdev->dev, "nvidia,egm-pxm", + pegmpxm); +} + static int nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev, struct nvgrace_gpu_pci_core_device *nvdev, @@ -1210,6 +1221,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, const struct vfio_device_ops *ops = &nvgrace_gpu_pci_core_ops; struct nvgrace_gpu_pci_core_device *nvdev; u64 memphys, memlength; + u64 egmpxm; int ret; ret = nvgrace_gpu_probe_check_device_ready(pdev); @@ -1217,9 +1229,14 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, return ret; ret = nvgrace_gpu_fetch_memory_property(pdev, &memphys, &memlength); - if (!ret) + if (!ret) { ops = &nvgrace_gpu_pci_ops; + ret = nvgrace_gpu_has_egm_property(pdev, &egmpxm); + if (!ret) + egm_enabled = true; + } + nvdev = vfio_alloc_device(nvgrace_gpu_pci_core_device, core_device.vdev, &pdev->dev, ops); if (IS_ERR(nvdev)) @@ -1239,6 +1256,12 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, if (ret) goto out_put_vdev; nvdev->core_device.pci_ops = &nvgrace_gpu_pci_dev_ops; + + if (egm_enabled) { + register_egm_node(pdev); + nvdev->egm_node = egmpxm; + } + } else { nvdev->core_device.pci_ops = &nvgrace_gpu_pci_dev_core_ops; } @@ -1257,6 +1280,12 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, static void nvgrace_gpu_remove(struct pci_dev *pdev) { struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); + struct nvgrace_gpu_pci_core_device *nvdev = + container_of(core_device, struct nvgrace_gpu_pci_core_device, + core_device); + + if (egm_enabled) + unregister_egm_node(nvdev->egm_node); vfio_pci_core_unregister_device(core_device); vfio_put_device(&core_device->vdev); From d769dad9c293d5e217018b30bb1c1c3bc0966394 Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Thu, 29 Aug 2024 08:15:41 +0000 Subject: [PATCH 02/19] NVIDIA: SAUCE: vfio/nvgrace-egm: Handle pages with ECC errors on the EGM BugLink: https://bugs.launchpad.net/bugs/2119656 It is possible for some system memory pages on the EGM to have uncorrectable ECC errors. A list of pages known with such errors (referred as retired pages) are maintained by the Host UEFI. The Host UEFI populates such list in a reserved region. It communicates the SPA of this region through a ACPI DSDT property. nvgrace-egm module is responsible to store the list of retired page offsets to be made available for usermode processes. The module: 1. Get the reserved memory region SPA and maps to it to fetch the list of bad pages. 2. Calculate the retired page offsets in the EGM and stores it. 3. Expose an ioctl to allow querying of the offsets. The ioctl is called by usermode apps such as QEMU to get the retired page offsets. The usermode apps are expected to take appropriate action to communicate the list to the VM. Signed-off-by: Ankit Agrawal Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit be54641b9f3e52a471e9d02aa12723bfb47a7060 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit c4cb1930d93ac2c7bb4f0cfba0a9e3e4ff180879 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 6b0a6d6644e3 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 64942451faf3b08f9f3014d70c871fe0ffcf041f noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit 4d66732ebda2be0d335d20f168d5317f7046e738 https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 126 +++++++++++++++++++++++++++++ include/uapi/linux/egm.h | 26 ++++++ 2 files changed, 152 insertions(+) create mode 100644 include/uapi/linux/egm.h diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index f3c22a9dfecb9..8c9ff6313e9f4 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -4,6 +4,8 @@ */ #include +#include +#include #include "egm.h" #define MAX_EGM_NODES 256 @@ -16,6 +18,12 @@ struct egm_region { size_t egmlength; struct device device; struct cdev cdev; + DECLARE_HASHTABLE(htbl, 0x10); +}; + +struct h_node { + unsigned long mem_offset; + struct hlist_node node; }; static dev_t dev; @@ -76,11 +84,80 @@ static int nvgrace_egm_mmap(struct file *file, struct vm_area_struct *vma) return ret; } +static long nvgrace_egm_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + unsigned long minsz = offsetofend(struct egm_bad_pages_list, count); + struct egm_bad_pages_list info; + void __user *uarg = (void __user *)arg; + struct egm_region *region = file->private_data; + + if (copy_from_user(&info, uarg, minsz)) + return -EFAULT; + + if (info.argsz < minsz) + return -EINVAL; + + if (!region) + return -EINVAL; + + switch (cmd) { + case EGM_BAD_PAGES_LIST: + int ret; + unsigned long bad_page_struct_size = sizeof(struct egm_bad_pages_info); + struct egm_bad_pages_info tmp; + struct h_node *cur_page; + struct hlist_node *tmp_node; + unsigned long bkt; + int count = 0, index = 0; + + hash_for_each_safe(region->htbl, bkt, tmp_node, cur_page, node) + count++; + + if (info.argsz < (minsz + count * bad_page_struct_size)) { + info.argsz = minsz + count * bad_page_struct_size; + info.count = 0; + goto done; + } else { + hash_for_each_safe(region->htbl, bkt, tmp_node, cur_page, node) { + /* + * This check fails if there was an ECC error + * after the usermode app read the count of + * bad pages through this ioctl. + */ + if (minsz + index * bad_page_struct_size >= info.argsz) { + info.argsz = minsz + index * bad_page_struct_size; + info.count = index; + goto done; + } + + tmp.offset = cur_page->mem_offset; + tmp.size = PAGE_SIZE; + + ret = copy_to_user(uarg + minsz + + index * bad_page_struct_size, + &tmp, bad_page_struct_size); + if (ret) + return ret; + index++; + } + + info.count = index; + } + break; + default: + return -EINVAL; + } + +done: + return copy_to_user(uarg, &info, minsz) ? -EFAULT : 0; +} + static const struct file_operations file_ops = { .owner = THIS_MODULE, .open = nvgrace_egm_open, .release = nvgrace_egm_release, .mmap = nvgrace_egm_mmap, + .unlocked_ioctl = nvgrace_egm_ioctl, }; static int setup_egm_chardev(struct egm_region *region) @@ -143,6 +220,45 @@ nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, return ret; } +static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, + struct egm_region *region) +{ + u64 retiredpagesphys, count; + void *memaddr; + int index; + + if (device_property_read_u64(&pdev->dev, + "nvidia,egm-retired-pages-data-base", + &retiredpagesphys)) + return; + + memaddr = memremap(retiredpagesphys, PAGE_SIZE, MEMREMAP_WB); + if (!memaddr) + return; + + count = *(u64 *)memaddr; + + hash_init(region->htbl); + + for (index = 0; index < count; index++) { + struct h_node *retired_page; + + /* + * Since the EGM is linearly mapped, the offset in the + * carveout is the same offset in the VM system memory. + * + * Calculate the offset to communicate to the usermode + * apps. + */ + retired_page = (struct h_node *)(vzalloc(sizeof(struct h_node))); + retired_page->mem_offset = *((u64 *)memaddr + index + 1) - + region->egmphys; + hash_add(region->htbl, &retired_page->node, retired_page->mem_offset); + } + + memunmap(memaddr); +} + int register_egm_node(struct pci_dev *pdev) { struct egm_region *region = NULL; @@ -165,6 +281,8 @@ int register_egm_node(struct pci_dev *pdev) atomic_set(®ion->open_count, 0); + nvgrace_egm_fetch_bad_pages(pdev, region); + list_add_tail(®ion->list, &egm_list); setup_egm_chardev(region); @@ -181,9 +299,17 @@ static void destroy_egm_chardev(struct egm_region *region) void unregister_egm_node(int egm_node) { struct egm_region *region, *temp_region; + struct h_node *cur_page; + unsigned long bkt; + struct hlist_node *temp_node; list_for_each_entry_safe(region, temp_region, &egm_list, list) { if (egm_node == region->egmpxm) { + hash_for_each_safe(region->htbl, bkt, temp_node, cur_page, node) { + hash_del(&cur_page->node); + vfree(cur_page); + } + destroy_egm_chardev(region); list_del(®ion->list); } diff --git a/include/uapi/linux/egm.h b/include/uapi/linux/egm.h new file mode 100644 index 0000000000000..8a808e45c2052 --- /dev/null +++ b/include/uapi/linux/egm.h @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#ifndef _UAPIEGM_H +#define _UAPIEGM_H + +#define EGM_TYPE ('E') + +struct egm_bad_pages_info { + __aligned_u64 offset; + __aligned_u64 size; +}; + +struct egm_bad_pages_list { + __u32 argsz; + /* out */ + __u32 count; + /* out */ + struct egm_bad_pages_info bad_pages[]; +}; + +#define EGM_BAD_PAGES_LIST _IO(EGM_TYPE, 100) + +#endif /* _UAPIEGM_H */ From 8f4d914eab7c79718346e9441239ffb32988d903 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 29 Aug 2024 18:49:03 -0700 Subject: [PATCH 03/19] NVIDIA: SAUCE: arm64: configs: Build CONFIG_NVGRACE_EGM as LKM BugLink: https://bugs.launchpad.net/bugs/2119656 Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit 5bb23c179220ec77ac9fb2ed610618ce1a902bd4 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 7d2ea5531c96fb9acd5704b6bec20aa29ca1fd39 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 077c8340953f noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit f5a03d00aed1fb83a77abd723dee5a8a79392f3b noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (backported from commit 48a4124e15a0c4ba9798a155ad45049fbda5207b https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) [kelseys: no changes to patch, resolved context conflict] Signed-off-by: Kelsey Steele --- arch/arm64/configs/defconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index 0e60e278c32cc..16d890daf1c2c 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -1824,3 +1824,4 @@ CONFIG_CORESIGHT_STM=m CONFIG_CORESIGHT_CPU_DEBUG=m CONFIG_CORESIGHT_CTI=m CONFIG_MEMTEST=y +CONFIG_NVGRACE_EGM=m From d744ddc8e7f4c2eb79760bee8f851798cebe7c2b Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Sun, 13 Oct 2024 04:53:38 +0000 Subject: [PATCH 04/19] NVIDIA: SAUCE: vfio/nvgrace-egm: Move the egm header file to include BugLink: https://bugs.launchpad.net/bugs/2119656 nvgrace-egm exposes the API register_egm_node & unregister_egm_node to manage EGM (Extended GPU Memory) present on the system. To allow out-of-tree driver such as nvidia-vgpu-vfio make use of them, move the declaration to a new nvgrace-egm.h in include. Signed-off-by: Ankit Agrawal Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit bed340f2023f22192893e9121834ee3ce252edd1 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit a9616639ce81799f5b3133c47c25f8d875728f4f https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 020c46c87e7a noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 739457a5ff1b0b30f904d2c1ab28eebcd670a628 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit 627a278af58878b302f3c2e610e02b796f1282d7 https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 2 +- drivers/vfio/pci/nvgrace-gpu/main.c | 2 +- .../pci/nvgrace-gpu/egm.h => include/linux/nvgrace-egm.h | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-) rename drivers/vfio/pci/nvgrace-gpu/egm.h => include/linux/nvgrace-egm.h (55%) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 8c9ff6313e9f4..598a1d07d00b7 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -6,7 +6,7 @@ #include #include #include -#include "egm.h" +#include #define MAX_EGM_NODES 256 diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index 0ef849da0e19c..cefa212f40833 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -10,7 +10,7 @@ #include #include #include -#include "egm.h" +#include /* * The device memory usable to the workloads running in the VM is cached diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.h b/include/linux/nvgrace-egm.h similarity index 55% rename from drivers/vfio/pci/nvgrace-gpu/egm.h rename to include/linux/nvgrace-egm.h index 28cc59e04a0b0..48add892aa5bf 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.h +++ b/include/linux/nvgrace-egm.h @@ -1,12 +1,12 @@ -// SPDX-License-Identifier: GPL-2.0-only +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ /* * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved */ -#ifndef NVGRACE_EGM_H -#define NVGRACE_EGM_H +#ifndef _NVGRACE_EGM_H +#define _NVGRACE_EGM_H int register_egm_node(struct pci_dev *pdev); void unregister_egm_node(int egm_node); -#endif /* NVGRACE_EGM_H */ +#endif /* _NVGRACE_EGM_H */ From d0e68bbed6d3dba71a086b7ffd048fd641d2cb81 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 15:06:57 -0800 Subject: [PATCH 05/19] NVIDIA: SAUCE: vfio/nvgrace-egm: Free region memory during unregistration BugLink: https://bugs.launchpad.net/bugs/2119656 Free the kmalloc'd region when the EGM is unregistered. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit fc592b9b4f8b455205abd2b2395671a831bb942e https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit f24760ccecb8c5517fca6791082ab89cf94b9f9f https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 374b166787e0 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 8f781d07d28638ab3c31c46ed79c0fdc9711a9c4 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit de1f033a5f4157c8d024401237b6710aa8a3c34d https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 598a1d07d00b7..06f41049275bd 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -312,6 +312,7 @@ void unregister_egm_node(int egm_node) destroy_egm_chardev(region); list_del(®ion->list); + kfree(region); } } } From fb1ea76a82c739d9e9ec30851826457d51ee50d0 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 15:38:11 -0800 Subject: [PATCH 06/19] NVIDIA: SAUCE: vfio/nvgrace-egm: Move region hash initialization BugLink: https://bugs.launchpad.net/bugs/2119656 Move region hash initiaization alongside the other region initialization statements to avoid situations where the hash table was not properly initialized. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit 8021c1d2b1c73015102bc69eda0029114989dd1f https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit e1264a62e8841fd5332f7f02a921242ff1b51dfa https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 0f8a09890f67 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 22f790add5eb4d1b17cd3e056d8f783af25b3f72 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit b0afaa851c6672445da78e278f8f98566d785640 https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 06f41049275bd..621d046084a18 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -238,8 +238,6 @@ static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, count = *(u64 *)memaddr; - hash_init(region->htbl); - for (index = 0; index < count; index++) { struct h_node *retired_page; @@ -279,6 +277,7 @@ int register_egm_node(struct pci_dev *pdev) region->egmlength = egmlength; region->egmpxm = egmpxm; + hash_init(region->htbl); atomic_set(®ion->open_count, 0); nvgrace_egm_fetch_bad_pages(pdev, region); From f8d9d144658a4c245b1dfdf3330f5abec728b7a9 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 15:48:47 -0800 Subject: [PATCH 07/19] NVIDIA: SAUCE: vfio/nvgrace-egm: Handle and convey EGM registration errors BugLink: https://bugs.launchpad.net/bugs/2119656 Update error handling within EGM regiration routine to catch and return errors to the caller. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit a57210c88c1c3693a24684c967c0858d75cabd32 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit a706ff8c445abed002e0b9493dfc9c664b1ffd57 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit edc0ac06e8e9 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit e7a177e2e55f20b7508e3017be2a5259a91e38d5 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit b8d20f648ee0fed893f0f4f55c4ff07c80f6d19e https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 621d046084a18..140f0f10f2c3e 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -273,6 +273,9 @@ int register_egm_node(struct pci_dev *pdev) } region = kvzalloc(sizeof(*region), GFP_KERNEL); + if (!region) + return -ENOMEM; + region->egmphys = egmphys; region->egmlength = egmlength; region->egmpxm = egmpxm; @@ -282,11 +285,16 @@ int register_egm_node(struct pci_dev *pdev) nvgrace_egm_fetch_bad_pages(pdev, region); - list_add_tail(®ion->list, &egm_list); + ret = setup_egm_chardev(region); + if (ret) + goto err; - setup_egm_chardev(region); + list_add_tail(®ion->list, &egm_list); return 0; +err: + kfree(region); + return ret; } EXPORT_SYMBOL_GPL(register_egm_node); From 6a5d4d3dae73a551eb152046115c53086bb59b01 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 15:55:58 -0800 Subject: [PATCH 08/19] NVIDIA: SAUCE: vfio/nvgrace-gpu: Handle EGM registration failure BugLink: https://bugs.launchpad.net/bugs/2119656 Detect and handle a failure from the EGM registration service. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit f18eee3bbdea77a9b525c0665d7ebe1992bb00b2 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 8371b68c33cc03a7ea6dfd7bdfc0fe9d47ec64fb https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit be5ae8ffa6ef noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 2dd59038740de38b4b2400930b05a49fd6051446 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit d841a67a5400d5070d265136bbaa4f6aec50a258 https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/main.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index cefa212f40833..8b8deab800a88 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -1258,7 +1258,10 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, nvdev->core_device.pci_ops = &nvgrace_gpu_pci_dev_ops; if (egm_enabled) { - register_egm_node(pdev); + ret = register_egm_node(pdev); + if (ret) + goto out_put_vdev; + nvdev->egm_node = egmpxm; } @@ -1268,10 +1271,13 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, ret = vfio_pci_core_register_device(&nvdev->core_device); if (ret) - goto out_put_vdev; + goto out_egm_unreg; return ret; +out_egm_unreg: + if (egm_enabled) + unregister_egm_node(nvdev->egm_node); out_put_vdev: vfio_put_device(&nvdev->core_device.vdev); return ret; From a8796469f5eb4e092287b496acabffa2bb42d808 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 16:07:26 -0800 Subject: [PATCH 09/19] NVIDIA: SAUCE: vfio/nvgrace-egm: Address sparse errors BugLink: https://bugs.launchpad.net/bugs/2119656 Fix minor syntax errors from sparse. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit bbb64e63a0b5e8c8eeec52b1e901745ba64b96d3 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit fe7819421a04be2b2405376da3550beb03986b6c https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit b19296004d0d noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit c1d3f2196850eef4c2eb77326d9475252e7a4087 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit 9145d3c5fcbc86b018d8b252d3f5353849fb8f86 https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 140f0f10f2c3e..33ed9a1f1a03f 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -102,6 +102,7 @@ static long nvgrace_egm_ioctl(struct file *file, unsigned int cmd, unsigned long switch (cmd) { case EGM_BAD_PAGES_LIST: + { int ret; unsigned long bad_page_struct_size = sizeof(struct egm_bad_pages_info); struct egm_bad_pages_info tmp; @@ -144,6 +145,7 @@ static long nvgrace_egm_ioctl(struct file *file, unsigned int cmd, unsigned long info.count = index; } break; + } default: return -EINVAL; } From 23ccee98ee4b8afc2171fd4dcdef0061f579a88d Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Fri, 22 Nov 2024 15:48:10 -0800 Subject: [PATCH 10/19] NVIDIA: SAUCE: vfio/nvgrace-egm: Ensure ACPI value reads are successful BugLink: https://bugs.launchpad.net/bugs/2119656 Ensure ACPI table reads are successful prior to using the value. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit b2947b075de6c887660cc8bc23ab5f0b6e7bfd17 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 92583550c3b22d1d00bfc6f59f3fc943cbd3a29e https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 2c5b472932c1 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 3f72f24ead3380c02907ee5d145f1b7a8e053d96 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit 63ccf200c039e454c06ee5312424444ccaa6ef7c https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 33ed9a1f1a03f..9388bdefe09aa 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -215,11 +215,13 @@ nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, ret = device_property_read_u64(&pdev->dev, "nvidia,egm-pxm", pegmpxm); + if (ret) + return ret; if (*pegmpxm > type_max(phys_addr_t)) return -EOVERFLOW; - return ret; + return 0; } static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, From 064a975ff718556948e3a0374463ede6a014fc09 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 14 Nov 2024 08:12:22 -0800 Subject: [PATCH 11/19] NVIDIA: SAUCE: vfio/nvgrace-egm: Avoid invalid retired pages base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BugLink: https://bugs.launchpad.net/bugs/2119656 Some environments may provide a "nvidia,egm-retired-pages-data-base” but fail to populate it with a base address, leaving it NULL. Mapping this invalid value results in a synchronous exception when the region is first touched. Detect a NULL value, generate a warning to draw attention to the firmware bug, and return without mapping. INFO: th500_ras_intr_handler: External Abort reason=1 syndrome=0x92000410 flags=0x1 [ 82.104493] Internal error: synchronous external abort: 0000000096000410 [#1] SMP [ 82.114898] Modules linked in: nvgrace_gpu_vfio_pci(E) nvgrace_egm(E) [ 82.257218] CPU: 0 PID: 10 Comm: kworker/0:1 Tainted: G OE 6.8.12+ #5 [ 82.265135] Hardware name: NVIDIA GH200 P5042, BIOS 24103110 20241031 [ 82.271720] Workqueue: events work_for_cpu_fn [ 82.276180] pstate: 03400009 (nzcv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--) [ 82.283298] pc : register_egm_node+0x2cc/0x440 [nvgrace_egm] [ 82.289087] lr : register_egm_node+0x2c4/0x440 [nvgrace_egm] [ 82.294872] sp : ffff8000802ebc30 [ 82.298254] x29: ffff8000802ebc60 x28: 00000000000000ff x27: 0000000000000000 [ 82.305550] x26: ffff000087a320c8 x25: ffff0000a5700000 x24: ffff000087a32000 [ 82.312846] x23: ffffa77cd758e368 x22: 0000000000000000 x21: ffffa77cd758c640 [ 82.320141] x20: ffffa77cd758e170 x19: ffff800081e7d000 x18: ffff800080293038 [ 82.327437] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 [ 82.334732] x14: 0000000000000000 x13: 65203a65646f6e5f x12: 0000000000000000 [ 82.342027] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000 [ 82.349322] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000 [ 82.356618] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000 [ 82.363913] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff800081e7d000 [ 82.371210] Call trace: [ 82.373705] register_egm_node+0x2cc/0x440 [nvgrace_egm] [ 82.379135] nvgrace_gpu_probe+0x2ac/0x528 [nvgrace_gpu_vfio_pci] [ 82.385366] local_pci_probe+0x4c/0xe0 [ 82.389198] work_for_cpu_fn+0x28/0x58 [ 82.393026] process_one_work+0x168/0x3f0 [ 82.397123] worker_thread+0x360/0x480 [ 82.400952] kthread+0x11c/0x128 [ 82.404248] ret_from_fork+0x10/0x20 [ 82.407906] Code: d2820001 940002b3 aa0003f3 b4fffac0 (f9400017) [ 82.414134] ---[ end trace 0000000000000000 ]--- Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit 7ba29302925c6f2e1b9825d06f7468acc175ab85 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 349fb1c23faef926f3bdbc479b088c7b6b66853f https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 6e9c94a06e83 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit c5992d5bdc0402f5c327a5fe65791cb5df278de9 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit aff377b1b9239f257ee47782fc6d01826954e077 https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 9388bdefe09aa..2ffae71f7f458 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -236,6 +236,10 @@ static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, &retiredpagesphys)) return; + /* Catch firmware bug and avoid a crash */ + if (WARN_ON_ONCE(retiredpagesphys == 0)) + return; + memaddr = memremap(retiredpagesphys, PAGE_SIZE, MEMREMAP_WB); if (!memaddr) return; From 9b0b922e414894726fa158d301a1b6e5b87d384d Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 23 Jan 2025 12:07:12 -0800 Subject: [PATCH 12/19] NVIDIA: SAUCE: vfio/nvgrace-egm: Update EGM unregistration API BugLink: https://bugs.launchpad.net/bugs/2119656 In an effort to simplify the programming model, use a symmetrical model for the the EGM regsiration APIs. This avoids the caller needing to keep a cookie or even have knowlege of if EGM is supported. Update the EGM unregisration API to use the PCI device as its parameter. Signed-off-by: Matthew R. Ochs (cherry picked from commit d8903ecbf6ae94cbf67b8492996021cd2488033c https://github.com/nvmochs/NV-Kernels/tree/vegm_01232025) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 5839fc506349c858a90a19e713c46fce025b2ec6 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit f6fb40e917fd noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 0e607bc7591e5b9e161ae98dfe9e42c02864b402 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit c89cfeae7480094d0c93c363aea8c9488b738124 https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 10 ++++++++-- drivers/vfio/pci/nvgrace-gpu/main.c | 4 ++-- include/linux/nvgrace-egm.h | 2 +- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 2ffae71f7f458..1545ac695ad77 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -311,15 +311,21 @@ static void destroy_egm_chardev(struct egm_region *region) cdev_device_del(®ion->cdev, ®ion->device); } -void unregister_egm_node(int egm_node) +void unregister_egm_node(struct pci_dev *pdev) { struct egm_region *region, *temp_region; struct h_node *cur_page; unsigned long bkt; struct hlist_node *temp_node; + u64 egmphys, egmlength, egmpxm; + int ret; + + ret = nvgrace_gpu_fetch_egm_property(pdev, &egmphys, &egmlength, &egmpxm); + if (ret) + return; list_for_each_entry_safe(region, temp_region, &egm_list, list) { - if (egm_node == region->egmpxm) { + if (egmpxm == region->egmpxm) { hash_for_each_safe(region->htbl, bkt, temp_node, cur_page, node) { hash_del(&cur_page->node); vfree(cur_page); diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index 8b8deab800a88..ab4de35b3860a 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -1277,7 +1277,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, out_egm_unreg: if (egm_enabled) - unregister_egm_node(nvdev->egm_node); + unregister_egm_node(pdev); out_put_vdev: vfio_put_device(&nvdev->core_device.vdev); return ret; @@ -1291,7 +1291,7 @@ static void nvgrace_gpu_remove(struct pci_dev *pdev) core_device); if (egm_enabled) - unregister_egm_node(nvdev->egm_node); + unregister_egm_node(pdev); vfio_pci_core_unregister_device(core_device); vfio_put_device(&core_device->vdev); diff --git a/include/linux/nvgrace-egm.h b/include/linux/nvgrace-egm.h index 48add892aa5bf..4bbd383a02732 100644 --- a/include/linux/nvgrace-egm.h +++ b/include/linux/nvgrace-egm.h @@ -7,6 +7,6 @@ #define _NVGRACE_EGM_H int register_egm_node(struct pci_dev *pdev); -void unregister_egm_node(int egm_node); +void unregister_egm_node(struct pci_dev *pdev); #endif /* _NVGRACE_EGM_H */ From 2ec85b3bafba42ce98733285d094adb49f44e9d5 Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Tue, 6 May 2025 09:38:38 -0500 Subject: [PATCH 13/19] NVIDIA: SAUCE: vfio/nvgrace-egm: track GPUs associated with the EGM regions BugLink: https://bugs.launchpad.net/bugs/2119656 GB200 systems could have multiple GPUs associated with an EGM region. For proper EGM functionality the host topology in terms of GPU affinity has to be replicated in the VM. Hence the EGM region structure must track the GPU devices belonging to the same socket. On the device probe, the device pci_dev struct is added to a linked list of the appropriate EGM region. Similarly on device remove, the pci_dev struct for the GPU is removed from the EGM region. Signed-off-by: Ankit Agrawal Ref: sj24: /home/nvidia/ankita/kernel_patches/0001_vfio_nvgrace-egm_track_GPUs_associated_with_the_EGM_regions.patch (koba: Enhance error handling, Remove egm_node from unregister_egm_node and move destroy_egm_chardev a little forward) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 0222c35fb26285ee1a6185ef50414093850ea352 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 5ba1a1f84f9d noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit c167095eba967b7ae64023927a015df0643c7460 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit 1f08cffb1bf0de24d3e94dc14097a8ba92bdc2a0 https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 68 ++++++++++++++++++++++++++---- 1 file changed, 59 insertions(+), 9 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 1545ac695ad77..67cc5254f681b 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -10,6 +10,11 @@ #define MAX_EGM_NODES 256 +struct gpu_node { + struct list_head list; + struct pci_dev *pdev; +}; + struct egm_region { struct list_head list; int egmpxm; @@ -18,6 +23,7 @@ struct egm_region { size_t egmlength; struct device device; struct cdev cdev; + struct list_head gpus; DECLARE_HASHTABLE(htbl, 0x10); }; @@ -187,6 +193,11 @@ static int setup_egm_chardev(struct egm_region *region) return ret; } +static void destroy_egm_chardev(struct egm_region *region) +{ + cdev_device_del(®ion->cdev, ®ion->device); +} + static int nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, u64 *pegmlength, u64 *pegmpxm) @@ -265,6 +276,32 @@ static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, memunmap(memaddr); } +static int add_gpu(struct egm_region *region, struct pci_dev *pdev) +{ + struct gpu_node *node; + + node = kvzalloc(sizeof(*node), GFP_KERNEL); + if (!node) + return -ENOMEM; + + node->pdev = pdev; + + list_add_tail(&node->list, ®ion->gpus); + return 0; +} + +static void remove_gpu(struct egm_region *region, struct pci_dev *pdev) +{ + struct gpu_node *node, *tmp; + + list_for_each_entry_safe(node, tmp, ®ion->gpus, list) { + if (node->pdev == pdev) { + list_del(&node->list); + kvfree(node); + } + } +} + int register_egm_node(struct pci_dev *pdev) { struct egm_region *region = NULL; @@ -275,11 +312,15 @@ int register_egm_node(struct pci_dev *pdev) if (ret) return ret; + /* Check if region already exists */ list_for_each_entry(region, &egm_list, list) { - if (region->egmphys == egmphys) - return 0; + if (region->egmphys == egmphys) { + /* Add GPU to existing region */ + return add_gpu(region, pdev); + } } + /* Create new region */ region = kvzalloc(sizeof(*region), GFP_KERNEL); if (!region) return -ENOMEM; @@ -289,28 +330,33 @@ int register_egm_node(struct pci_dev *pdev) region->egmpxm = egmpxm; hash_init(region->htbl); + INIT_LIST_HEAD(®ion->gpus); + atomic_set(®ion->open_count, 0); nvgrace_egm_fetch_bad_pages(pdev, region); ret = setup_egm_chardev(region); if (ret) - goto err; + goto err_free_region; list_add_tail(®ion->list, &egm_list); + ret = add_gpu(region, pdev); + if (ret) + goto err_remove_from_list; + return 0; -err: + +err_remove_from_list: + list_del(®ion->list); + destroy_egm_chardev(region); +err_free_region: kfree(region); return ret; } EXPORT_SYMBOL_GPL(register_egm_node); -static void destroy_egm_chardev(struct egm_region *region) -{ - cdev_device_del(®ion->cdev, ®ion->device); -} - void unregister_egm_node(struct pci_dev *pdev) { struct egm_region *region, *temp_region; @@ -326,6 +372,10 @@ void unregister_egm_node(struct pci_dev *pdev) list_for_each_entry_safe(region, temp_region, &egm_list, list) { if (egmpxm == region->egmpxm) { + remove_gpu(region, pdev); + if (!list_empty(®ion->gpus)) + break; + hash_for_each_safe(region->htbl, bkt, temp_node, cur_page, node) { hash_del(&cur_page->node); vfree(cur_page); From 34a44e55dfb72a637120fc233024afbb150d30df Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Tue, 6 May 2025 09:39:33 -0500 Subject: [PATCH 14/19] NVIDIA: SAUCE: vfio/nvgrace-egm: list gpus through sysfs BugLink: https://bugs.launchpad.net/bugs/2119656 To replicate the host EGM topology in the VM in terms of the GPU affinity, the userspace need to be aware of which GPUs belong to the same socket as the EGM region. Expose the list of GPUs associated with an EGM region through sysfs. The list can be queried from the location /sys/devices/virtual/egm/egmX/gpu_devices. Signed-off-by: Ankit Agrawal Ref: sj24: /home/nvidia/ankita/kernel_patches/0002_vfio_nvgrace-egm_list_gpus_through_sysfs.patch (koba: Enchance error handling for sysfs_create_group) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit fec2356d20f7054c0c89b1d32e7862bba34bda54 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 5dde2f0e0bb5 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit a5284ca673f5a3732431c7e1cb49a48c347a0087 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit cc460ff0d7bde7598baf087bfd9e59835d20e32f https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 41 +++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 67cc5254f681b..2988d55208bf4 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -276,6 +276,38 @@ static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, memunmap(memaddr); } +static ssize_t gpu_devices_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct egm_region *region = + container_of(dev, struct egm_region, device); + struct gpu_node *node, *temp_node; + int len = 0; + + list_for_each_entry_safe(node, temp_node, ®ion->gpus, list) { + struct pci_dev *pdev = node->pdev; + + len += sysfs_emit_at(buf, len, "%04x:%02x:%02x.%x\n", + pci_domain_nr(pdev->bus), + pdev->bus->number, + PCI_SLOT(pdev->devfn), + PCI_FUNC(pdev->devfn)); + } + + return len; +} + +static DEVICE_ATTR_RO(gpu_devices); + +static struct attribute *attrs[] = { + &dev_attr_gpu_devices.attr, + NULL, +}; + +static struct attribute_group attr_group = { + .attrs = attrs, +}; + static int add_gpu(struct egm_region *region, struct pci_dev *pdev) { struct gpu_node *node; @@ -342,12 +374,18 @@ int register_egm_node(struct pci_dev *pdev) list_add_tail(®ion->list, &egm_list); - ret = add_gpu(region, pdev); + ret = sysfs_create_group(®ion->device.kobj, &attr_group); if (ret) goto err_remove_from_list; + ret = add_gpu(region, pdev); + if (ret) + goto err_remove_sysfs; + return 0; +err_remove_sysfs: + sysfs_remove_group(®ion->device.kobj, &attr_group); err_remove_from_list: list_del(®ion->list); destroy_egm_chardev(region); @@ -381,6 +419,7 @@ void unregister_egm_node(struct pci_dev *pdev) vfree(cur_page); } + sysfs_remove_group(®ion->device.kobj, &attr_group); destroy_egm_chardev(region); list_del(®ion->list); kfree(region); From 3555f607ef63dff37f5d5ce034119cd043671da4 Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Tue, 6 May 2025 09:40:16 -0500 Subject: [PATCH 15/19] NVIDIA: SAUCE: vfio/nvgrace-egm: expose the egm size through sysfs BugLink: https://bugs.launchpad.net/bugs/2119656 To allocate the EGM, the userspace need to know it's size. Currently, there is no easy way for the userspace to determine that. Make nvgrace-egm expose the size through sysfs that can be queried by the userspace from /sys/devices/virtual/egm/egmX/egm_size. Signed-off-by: Ankit Agrawal Ref: sj24: /home/nvidia/ankita/kernel_patches/0003_vfio_nvgrace-egm_expose_the_egm_size_through_sysfs.patch Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit dcdcef245e8d648d38ef75f1023c7437b5639ddf https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 994015745197 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit e025c29ebeb237a2e85b86e5c6b1f83602e7f8bd noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit 4bf52ef026e64eed8b69498a9302583562cfd2a0 https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 2988d55208bf4..1e8f2f10b06f9 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -299,8 +299,19 @@ static ssize_t gpu_devices_show(struct device *dev, struct device_attribute *att static DEVICE_ATTR_RO(gpu_devices); +static ssize_t egm_size_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct egm_region *region = + container_of(dev, struct egm_region, device); + return sysfs_emit(buf, "0x%lx\n", region->egmlength); +} + +static DEVICE_ATTR_RO(egm_size); + static struct attribute *attrs[] = { &dev_attr_gpu_devices.attr, + &dev_attr_egm_size.attr, NULL, }; From 4d1cbca8d9b3f75f3d6140298fa92ea32fc91419 Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Sun, 18 Jan 2026 02:03:13 +0000 Subject: [PATCH 16/19] NVIDIA: SAUCE: vfio/nvgrace-egm: register EGM PFNMAP range with memory_failure BugLink: https://bugs.launchpad.net/bugs/2138892 EGM carveout memory is mapped directly into userspace (QEMU) and is not added to the kernel. It is not managed by the kernel page allocator and has no struct pages. The module can thus utilize the Linux memory manager's memory_failure mechanism for regions with no struct pages. The Linux MM code exposes register/unregister APIs allowing modules to register such memory regions for memory_failure handling. Register the EGM PFN range with the MM memory_failure infrastructure on open, and unregister it on the last close. Provide a PFN-to-VMA offset callback that validates the PFN is within the EGM region and the VMA, then converts it to a file offset and records the poisoned offset in the existing hashtable for reporting to userspace. Signed-off-by: Ankit Agrawal Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 3fde504ffb6ab8def2971607833069daa29835c4 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit de5a032dc34cc71a823ff84a12c4c87da0e648e6 https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 100 ++++++++++++++++++++++++++++- 1 file changed, 99 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 1e8f2f10b06f9..9e47813f4ecbe 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -7,6 +7,8 @@ #include #include #include +#include +#include #define MAX_EGM_NODES 256 @@ -25,6 +27,7 @@ struct egm_region { struct cdev cdev; struct list_head gpus; DECLARE_HASHTABLE(htbl, 0x10); + struct pfn_address_space pfn_address_space; }; struct h_node { @@ -36,11 +39,97 @@ static dev_t dev; static struct class *class; static struct list_head egm_list; +static int pfn_memregion_offset(struct egm_region *region, + unsigned long pfn, + pgoff_t *pfn_offset_in_region) +{ + unsigned long start_pfn, num_pages; + + start_pfn = PHYS_PFN(region->egmphys); + num_pages = region->egmlength >> PAGE_SHIFT; + + if (pfn < start_pfn || pfn >= start_pfn + num_pages) + return -EFAULT; + + *pfn_offset_in_region = pfn - start_pfn; + + return 0; +} + +static int track_ecc_offset(struct egm_region *region, + unsigned long mem_offset) +{ + struct h_node *cur_page, *ecc_page; + unsigned long bkt; + + hash_for_each(region->htbl, bkt, cur_page, node) { + if (cur_page->mem_offset == mem_offset) + return 0; + } + + ecc_page = (struct h_node *)(vzalloc(sizeof(struct h_node))); + if (!ecc_page) + return -ENOMEM; + + ecc_page->mem_offset = mem_offset; + + hash_add(region->htbl, &ecc_page->node, ecc_page->mem_offset); + + return 0; +} + +static int nvgrace_egm_pfn_to_vma_pgoff(struct vm_area_struct *vma, + unsigned long pfn, + pgoff_t *pgoff) +{ + struct egm_region *region = vma->vm_file->private_data; + pgoff_t vma_offset_in_region = vma->vm_pgoff & + ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1); + pgoff_t pfn_offset_in_region; + int ret; + + ret = pfn_memregion_offset(region, pfn, &pfn_offset_in_region); + if (ret) + return ret; + + /* Ensure PFN is not before VMA's start within the region */ + if (pfn_offset_in_region < vma_offset_in_region) + return -EFAULT; + + /* Calculate offset from VMA start */ + *pgoff = vma->vm_pgoff + + (pfn_offset_in_region - vma_offset_in_region); + + /* Track and save the poisoned offset */ + return track_ecc_offset(region, *pgoff << PAGE_SHIFT); +} + +static int +nvgrace_egm_vfio_pci_register_pfn_range(struct inode *inode, + struct egm_region *region) +{ + int ret; + unsigned long pfn, nr_pages; + + pfn = PHYS_PFN(region->egmphys); + nr_pages = region->egmlength >> PAGE_SHIFT; + + region->pfn_address_space.node.start = pfn; + region->pfn_address_space.node.last = pfn + nr_pages - 1; + region->pfn_address_space.mapping = inode->i_mapping; + region->pfn_address_space.pfn_to_vma_pgoff = nvgrace_egm_pfn_to_vma_pgoff; + + ret = register_pfn_address_space(®ion->pfn_address_space); + + return ret; +} + static int nvgrace_egm_open(struct inode *inode, struct file *file) { void *memaddr; struct egm_region *region = container_of(inode->i_cdev, struct egm_region, cdev); + int ret; if (!region) return -EINVAL; @@ -58,6 +147,12 @@ static int nvgrace_egm_open(struct inode *inode, struct file *file) memunmap(memaddr); file->private_data = region; + ret = nvgrace_egm_vfio_pci_register_pfn_range(inode, region); + if (ret && ret != -EOPNOTSUPP) { + file->private_data = NULL; + return ret; + } + return 0; } @@ -69,8 +164,11 @@ static int nvgrace_egm_release(struct inode *inode, struct file *file) if (!region) return -EINVAL; - if (atomic_dec_and_test(®ion->open_count)) + if (atomic_dec_and_test(®ion->open_count)) { + unregister_pfn_address_space(®ion->pfn_address_space); + file->private_data = NULL; + } return 0; } From 7f8c1e0f559fd25cb174cfda830cb478df186b09 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 20:03:50 -0800 Subject: [PATCH 17/19] NVIDIA: SAUCE: vfio/nvgrace-egm: Address smatch errors BugLink: https://bugs.launchpad.net/bugs/2119656 Return the intended errno upon a copyout fault, remove unnecessary checks following container_of pointer derivation, and use the correct macro and types for overflow checking. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit 429910b6fba450a9831f590d9622d16b79006311 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit bda63f340176a3a610a64512176f0e133b9efd9f https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 942bf3b26275 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 63dd05905701cd6f68b3194afafca2bedb344222 noble:linux-nvidia-6.17) Signed-off-by: Matthew R. Ochs (cherry picked from commit f8c59b75087fb94def8226043b9b19a77231b5a3 https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 9e47813f4ecbe..aa9e796b6fb4f 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -131,9 +131,6 @@ static int nvgrace_egm_open(struct inode *inode, struct file *file) struct egm_region, cdev); int ret; - if (!region) - return -EINVAL; - if (atomic_inc_return(®ion->open_count) > 1) return 0; @@ -161,9 +158,6 @@ static int nvgrace_egm_release(struct inode *inode, struct file *file) struct egm_region *region = container_of(inode->i_cdev, struct egm_region, cdev); - if (!region) - return -EINVAL; - if (atomic_dec_and_test(®ion->open_count)) { unregister_pfn_address_space(®ion->pfn_address_space); @@ -242,7 +236,7 @@ static long nvgrace_egm_ioctl(struct file *file, unsigned int cmd, unsigned long index * bad_page_struct_size, &tmp, bad_page_struct_size); if (ret) - return ret; + return -EFAULT; index++; } @@ -311,7 +305,7 @@ nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, if (ret) return ret; - if (*pegmlength > type_max(size_t)) + if (overflows_type(*pegmlength, size_t)) return -EOVERFLOW; ret = device_property_read_u64(&pdev->dev, "nvidia,egm-base-pa", @@ -319,7 +313,7 @@ nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, if (ret) return ret; - if (*pegmphys > type_max(phys_addr_t)) + if (overflows_type(*pegmphys, phys_addr_t)) return -EOVERFLOW; ret = device_property_read_u64(&pdev->dev, "nvidia,egm-pxm", @@ -327,7 +321,7 @@ nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, if (ret) return ret; - if (*pegmpxm > type_max(phys_addr_t)) + if (overflows_type(*pegmpxm, int)) return -EOVERFLOW; return 0; From 65b2930d9527e4ca672a5afbc67a6ed132e79fa5 Mon Sep 17 00:00:00 2001 From: kobakonvidia Date: Mon, 26 May 2025 16:48:35 +0000 Subject: [PATCH 18/19] NVIDIA: SAUCE: vfio/nvgrace-egm: Add null pointer checks after memory allocations BugLink: https://bugs.launchpad.net/bugs/2119656 Add missing null pointer checks after vzalloc() calls in the NVIDIA Grace GPU driver's EGM (External GPU Memory) handling code. This prevents potential null pointer dereferences in the memory failure handling and bad page fetching functions, providing proper error handling for allocation failures. Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 63127e2996a244841309ee86b4535f41e2b0de1f https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit e5f0c8d1ba27 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (backported from commit 862ed5a2b5c58c1a1bc5d6a9d012ea82ee177389 noble:linux-nvidia-6.17) [mochs: Addressed collission for a null pointer check that is no longer needed] Signed-off-by: Matthew R. Ochs (cherry picked from commit 8e8763e27a5e0de6559e76324430fb1e0184cd6c https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index aa9e796b6fb4f..9ad08c9cb59d0 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -360,6 +360,8 @@ static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, * apps. */ retired_page = (struct h_node *)(vzalloc(sizeof(struct h_node))); + if (!retired_page) + continue; /* Skip this entry on allocation failure */ retired_page->mem_offset = *((u64 *)memaddr + index + 1) - region->egmphys; hash_add(region->htbl, &retired_page->node, retired_page->mem_offset); From c5b8b947ec0e0e44c8087a8bd0512526e9ecdd1d Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Fri, 13 Feb 2026 04:11:25 +0000 Subject: [PATCH 19/19] NVIDIA: SAUCE: vfio/nvgrace-egm: split zapping EGM into 1GB chunks BugLink: https://bugs.launchpad.net/bugs/2142160 When initializing EGM (Extended GPU Memory) regions, the current implementation performs a single memset operation over the entire memory region. For very large regions, this can result in long-running uninterruptible operations that may cause system responsiveness issues or trigger watchdog timeouts. Split the memset operation into 1GB chunks. Signed-off-by: Ankit Agrawal Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (cherry picked from commit 355474478031a909b2588458efb9d2c48e571735 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin (cherry picked from commit a577394e804a57dcc6b21b9aba751f16732c777b https://github.com/NVIDIA/NV-Kernels/tree/26.04_linux-nvidia) Signed-off-by: Kelsey Steele --- drivers/vfio/pci/nvgrace-gpu/egm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 9ad08c9cb59d0..a2e4c05a83c34 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -140,7 +140,20 @@ static int nvgrace_egm_open(struct inode *inode, struct file *file) return -EINVAL; } - memset((u8 *)memaddr, 0, region->egmlength); + { + size_t remaining = region->egmlength; + u8 *chunk_addr = (u8 *)memaddr; + size_t chunk_size; + + while (remaining > 0) { + chunk_size = min(remaining, SZ_1G); + memset(chunk_addr, 0, chunk_size); + cond_resched(); + chunk_addr += chunk_size; + remaining -= chunk_size; + } + } + memunmap(memaddr); file->private_data = region;