NVIDIA GPU MIG Partitioning Guide

Maximize GPU utilization and reduce infrastructure costs

The process of GPU MIG (Multi-Instance GPU) partitioning is a vital step in optimizing GPU resource utilization. This guide covers the steps for configuring MIG partitioning on NVIDIA GPUs using Kubernetes. This method ensures efficient distribution of GPU workloads across multiple instances.

Before diving into MIG partitioning, ensure your GPU operator is deployed correctly. Here’s a detailed deployment guide in my YouTube video:

What is MIG Partitioning?

MIG allows splitting a single NVIDIA GPU into multiple independent GPU instances. Using MIG enables workloads to run on separate GPU partitions, enhancing resource management and utilization in environments like Kubernetes.

Requirements

Before beginning the partitioning process, the following components are necessary:

• A node with a MIG-capable NVIDIA GPU.
• Kubernetes cluster with GPU operator deployed.
• NVIDIA drivers installed and running.

Step 1: Label the Node

To configure MIG partitioning, you must label the node. Use the following command to label the node with the MIG configuration:

kubectl label nodes <node-name> nvidia.com/mig.config=all-1g.24gb --overwrite

Note: You can select the MIG configuration based on your specific requirements. For a detailed list of available MIG profiles, including memory fractions and hardware units, refer to the official NVIDIA MIG Profile Documentation.

Step 2: Verifying the GPU Partitioning

Once the label is applied, GPU partitioning can be verified. To check the status of the GPU and view the partitions, use the following command:

kubectl exec -it -n gpu-operator ds/nvidia-driver-daemonset -- nvidia-smi -L

This command lists all GPU partitions, confirming the successful configuration of MIG.

Step 3: Monitoring GPU Usage

After partitioning, it is important to monitor GPU utilization. MIG provides a clear division of GPU resources, allowing each instance to be monitored separately. Use nvidia-smi to check the status and usage metrics for each partition.

nvidia-smi

The system will display a detailed breakdown of each partition’s memory, usage, and performance.

Advantages of MIG Partitioning

• Improved GPU Utilization: Multiple workloads can share a single GPU, preventing underutilization.
• Better Isolation: Each partition operates independently, reducing resource conflicts.
• Optimized Workloads: The system allocates GPU resources based on the needs of individual tasks, maximizing efficiency..

MIG partitioning provides an efficient way to manage GPU resources, especially in environments with multiple workloads. By following these steps, users can ensure that their GPUs are optimally configured for better performance.

Enabling MIG Partitioning at GPU Operator Deployment

You must enable MIG partitioning during the deployment process when deploying the GPU operator. This can be configured by setting the MIG mode in the values.yaml file of the GPU operator.

You can use the following Helm command to enable MIG at the time of deployment:

helm install --wait  --generate-name -n gpu-operator --create-namespace nvidia/gpu-operator  --set mig.strategy=mixed  --set migManager.enabled=false

---

This ensures the GPU operator enables MIG partitioning on all managed GPUs.

---

For more details on GPU operator deployment, refer to my previous blog on NVIDIA GPU Deployment for AI in Kubernetes.