# nvidia-smi commands for monitoring GPU servers

> nvidia-smi monitoring commands to monitor GPU usage on Linux. Live views, CSV logs, nvtop, DCGM exporter, ECC and Xid errors, plus rocm-smi equivalents.

Source: https://overgpu.com/docs/gpu-monitoring  
Updated: 2026-10-22  
Publisher: OverGPU (https://overgpu.com/)

These nvidia-smi monitoring commands cover what you need to monitor GPU usage on Linux, from a live view during a training run to metrics you keep for weeks. They work the same on single-GPU plans and 8-GPU servers. The last sections cover hardware errors, when to open a ticket, and the AMD equivalents.

## nvidia-smi cheat sheet

| Task | Command |
| --- | --- |
| One snapshot of every GPU | `nvidia-smi` |
| Refresh every 2 seconds | `nvidia-smi -l 2` |
| List GPUs and their UUIDs | `nvidia-smi -L` |
| Stream use, power, clocks and memory | `nvidia-smi dmon -s pucm` |
| Processes and the memory they use | `nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv` |
| Chosen metrics as CSV, every 10 seconds | `nvidia-smi --query-gpu=index,utilization.gpu,memory.used --format=csv -l 10` |
| Temperature, power or ECC details | `nvidia-smi -q -d TEMPERATURE` (or `POWER`, `ECC`) |
| How GPUs connect: NVLink, PCIe, NUMA | `nvidia-smi topo -m` |
| NVLink status per link | `nvidia-smi nvlink -s` |

Each command is explained below, with the AMD equivalents at the end of the page.

## Watch GPUs live

```
nvidia-smi              # one snapshot
nvidia-smi -l 2         # print a new snapshot every 2 seconds
watch -n 2 nvidia-smi   # same, redrawn in place
```

**GPU-Util** is the share of time at least one kernel was running, not how busy every core is, so 100% does not prove the GPU is used efficiently. **Memory-Usage** is allocated memory; frameworks such as vLLM reserve most of it on purpose. **Pwr:Usage/Cap** and **Temp** show power draw and temperature, and **Perf** the performance state, P0 being the highest. The process list shows which PIDs use each GPU; inside containers it is often empty, so check from the host. If nvidia-smi itself fails, see [driver troubleshooting](https://overgpu.com/docs/nvidia-drivers-cuda).

## Stream metrics with dmon

```
nvidia-smi dmon -s pucm -d 5
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
```

`dmon` prints one line per GPU every 5 seconds (`-d 5`). The `-s` letters choose the metric groups: `p` power and temperature, `u` utilization, `c` clocks and `m` memory; add `e` for ECC and PCIe errors, `t` for PCIe throughput. `-i 0,1` limits it to some GPUs. The second command lists each process with the GPU memory it holds.

## Log to CSV with --query-gpu

```
fields=timestamp,index,utilization.gpu,memory.used,memory.total
fields=$fields,temperature.gpu,power.draw,clocks.sm
nvidia-smi --query-gpu=$fields --format=csv -l 10 >> ~/gpu-metrics.csv
```

This appends one line per GPU every 10 seconds; run it in tmux or as a service. Use `--format=csv,noheader,nounits` for files you parse with scripts. `nvidia-smi --help-query-gpu` lists every field, such as `temperature.memory` on HBM GPUs.

## nvtop, an htop for GPUs

```
sudo apt install -y nvtop
nvtop
```

nvtop draws utilization and memory charts for each GPU above a process list. Recent versions, such as the one in Ubuntu 24.04, also support AMD GPUs. Press Q to quit.

## Long-term metrics with DCGM exporter and Prometheus

For history and alerts, NVIDIA's DCGM exporter publishes GPU metrics in Prometheus format, and Grafana can chart them. A minimal start with Docker, using the current tag from NVIDIA's NGC catalog in place of `TAG`:

```
docker run -d --name dcgm-exporter --gpus all --cap-add SYS_ADMIN \
  -p 127.0.0.1:9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:TAG
curl -s http://127.0.0.1:9400/metrics \
  | grep -E 'DCGM_FI_DEV_(GPU_UTIL|GPU_TEMP|POWER_USAGE|XID_ERRORS)'
```

You should see one value per GPU for each metric. Scrape it from your Prometheus server over [private networking](https://overgpu.com/products/private-networking) or an SSH tunnel, and alert on `DCGM_FI_DEV_XID_ERRORS` and on temperature. Never publish the metrics port to the internet. On AMD servers, AMD's device metrics exporter for Instinct GPUs plays the same role.

## Temperature, power and ECC errors

```
nvidia-smi -q -d TEMPERATURE
nvidia-smi -q -d POWER,PERFORMANCE
nvidia-smi -q -d ECC
nvidia-smi -q -d ROW_REMAPPER
```

- **Temperature.** Compare the current reading with the "GPU Slowdown Temp" and "GPU Shutdown Temp" thresholds in the same output. Readings near the slowdown threshold reduce clocks.
- **Power and clocks.** The PERFORMANCE section lists clock event reasons. "SW Power Cap" under heavy load is normal; "HW Slowdown" or "HW Thermal Slowdown" while the GPU is busy is worth reporting.
- **ECC.** Data center GPUs correct single-bit memory errors on the fly. Volatile counters reset when the driver reloads; aggregate counters cover the GPU's lifetime. Uncorrectable errors usually stop the running job.
- **Row remapping.** HBM GPUs replace faulty memory rows. "Pending: Yes" means the GPU needs a reset or reboot to apply the remap. Repeated remaps or a remapping failure mean the GPU needs attention.

## Xid errors and when to open a ticket

The NVIDIA driver reports GPU faults in the kernel log as Xid messages:

```
sudo dmesg -T | grep -i xid
sudo journalctl -k --since "24 hours ago" | grep -i xid
```

| Xid | Meaning | What to do |
| --- | --- | --- |
| 13, 31, 43 | Graphics engine exception, GPU memory page fault, GPU stopped processing | Usually a bug or invalid memory access in the application. Open a ticket if they continue across different jobs. |
| 48, 94, 95 | Double-bit ECC error; contained or uncontained ECC error | Open a ticket. A contained error (94) stops only the affected application; after 95 the GPU must be reset. |
| 63, 64 | Row remapping or page retirement event, or a failure to record it | Reboot to apply a pending remap. Open a ticket if it repeats or fails. |
| 74 | NVLink error | Open a ticket with the full log line. |
| 79 | GPU has fallen off the bus | Open a ticket. The GPU stays unavailable until the server is rebooted, and may need repair. |

In your ticket, include the server IP, the Xid lines and an NVIDIA bug report, which gathers the relevant logs in one file:

```
sudo nvidia-bug-report.sh
```

It writes `nvidia-bug-report.log.gz` to the current directory. Failed GPUs are replaced as a priority under our [service level agreement](https://overgpu.com/legal/sla), and engineers answer 24/7: [open a support ticket](https://overgpu.com/help/getting-support) from your dashboard.

## AMD equivalents with rocm-smi and amd-smi

| Task | NVIDIA | AMD |
| --- | --- | --- |
| Snapshot | `nvidia-smi` | `rocm-smi` or `amd-smi metric` |
| Live view | `nvidia-smi -l 2` | `watch -n 2 rocm-smi` |
| Streaming table | `nvidia-smi dmon` | `amd-smi monitor` |
| Use, memory, power, temperature | `nvidia-smi --query-gpu=...` | `rocm-smi --showuse --showmemuse --showpower --showtemp` |
| Processes | `nvidia-smi --query-compute-apps=...` | `amd-smi process` |
| Kernel messages | `sudo dmesg -T \| grep -i xid` | `sudo dmesg -T \| grep -i amdgpu` |

More on the AMD tools in [ROCm setup](https://overgpu.com/docs/amd-rocm).

## Related pages

- [Check CUDA version and NVIDIA driver on Ubuntu](https://overgpu.com/docs/nvidia-drivers-cuda): Check NVIDIA driver and CUDA version with nvidia-smi and nvcc, see how they must match and add a CUDA toolkit on Ubuntu 24.04 without a new driver.
- [NCCL tests on a multi-GPU server: all_reduce_perf step by step](https://overgpu.com/docs/multi-gpu-nccl): Check NVLink topology, build nccl-tests, read busbw from all_reduce_perf and launch torchrun jobs across 8 GPUs.
- [GPU server SLA credits: how they work](https://overgpu.com/help/sla-credits): 99.9% monthly availability, credits of 10% to 100% of the monthly fee, what counts as downtime and how to claim one.
- [GPU server support: how tickets work](https://overgpu.com/help/getting-support): GPU server support at OverGPU runs 24/7 through dashboard tickets handled by engineers. See who can open one, what to include and what needs no ticket.