GPU server OS images with CUDA, ROCm and Docker

GPU server OS images at OverGPU come with the GPU stack already in place. NVIDIA servers boot Ubuntu 22.04 or 24.04 with NVIDIA drivers, CUDA, Docker and the NVIDIA Container Toolkit preinstalled, and AMD servers get ROCm images. You log in as root over SSH and start with your own code instead of a driver install.

What is in our GPU server OS images

We do not list fixed driver or CUDA version numbers on this page, because they move with GPU generations and releases. After login, nvidia-smi shows the driver version and the highest CUDA version that driver supports. If your project depends on a specific driver branch, mention it in the order notes, or install it yourself with root access.

On single-GPU plans, the image runs inside your dedicated VM, with the whole GPU passed through. On 8-GPU servers, it runs directly on the bare-metal machine. The checks below work the same way on both; only the topology output differs.

The Ubuntu CUDA image in practice

Pick 24.04 for a newer kernel and toolchain. Pick 22.04 if your build scripts or base containers still target it. Then run three checks:

nvidia-smi
nvidia-smi topo -m
docker run --rm --gpus all ubuntu nvidia-smi

The first lists every GPU with its memory, utilization and driver. The second prints the topology, and on an HGX server it shows NVLink between all eight GPUs. The third proves that containers reach the GPU through the NVIDIA Container Toolkit, which mounts the driver libraries and devices into the container when it starts.

Containers are the easiest way to keep CUDA versions under control. A framework image brings its own CUDA libraries, and the host only needs a driver recent enough for them. If an image needs a newer CUDA than the host driver supports, it typically refuses to start, so compare against the CUDA version shown by nvidia-smi first. Our Docker and containers guide and NVIDIA drivers and CUDA guide go further.

If you install frameworks directly instead, use one virtual environment per project and pick wheels built for a CUDA version your driver supports. PyTorch, for example, publishes builds for several CUDA versions, so the same rule applies: check nvidia-smi first, then choose the build.

ROCm images for AMD Instinct

AMD Instinct MI355X servers get ROCm images instead of CUDA. ROCm is AMD's software stack for GPU compute, and frameworks such as PyTorch and vLLM support it. Check the GPUs with rocm-smi, or amd-smi on recent ROCm releases, and list the devices with rocminfo. On an MI355X server, both should show all eight GPUs. Containers built for ROCm reach the GPUs through the /dev/kfd and /dev/dri devices, passed with --device flags.

Code that stays at the framework level usually moves over without changes. Custom CUDA kernels need porting with HIP. The AMD ROCm guide covers the details.

Other distributions, root access and reinstalls

Debian or Rocky Linux are available on request, for teams whose tooling, hardening baselines or configuration management already target one of them. Mention the distribution in the order notes for our engineers, together with what you expect to find on the image.

Every image gives you root over SSH: you choose the image, then log in with an SSH key or a root password, both set in the order form when you deploy a server. The SSH access guide covers keys and first login. You can install, upgrade or remove anything allowed by our acceptable use policy, including a different driver branch or container runtime.

For a clean start or a different image, open a support ticket from your dashboard to ask for a reinstall, at any hour. Back up what you need first: local storage is not backed up by us unless agreed in writing.

Preinstalled software, at no extra cost

To start from a ready application rather than a bare system, pick one in the order form when you deploy a server. Our engineers install and test it on your image before delivery, at no extra cost, and the note that comes with your server explains how to open it.

• Serve and chat with models: vLLM, SGLang, Ollama, Open WebUI with Ollama, llama.cpp, Text Generation WebUI, Hugging Face Text Generation Inference and NVIDIA Triton Inference Server, plus Langflow to build agents.
• Train and fine-tune: PyTorch or TensorFlow with JupyterLab, Unsloth and Axolotl.
• Images and video: ComfyUI, Stable Diffusion WebUI Forge, AUTOMATIC1111, InvokeAI, Fooocus, SwarmUI and Wan2GP, plus Kohya’s GUI, AI Toolkit and FluxGym to train LoRAs.
• Notebooks, code and more: JupyterLab, code-server, Whisper, NVIDIA RAPIDS, Blender and a remote Linux desktop.

Some of them run on NVIDIA GPUs only, and the order form shows only what fits the server you chose. The software stays yours to update, configure or remove.

Keep your stack reproducible

• Pin framework containers by tag or digest rather than latest.
• Hold driver packages during long training runs, for example with apt-mark hold, and upgrade between runs.
• Keep your setup in a script or a Dockerfile, so a reinstall or a second server goes through the same steps.
• Save the output of nvidia-smi with each experiment, so results can be traced to a driver version.
• After a kernel or driver update, reboot and run nvidia-smi before you start a long job, not after it fails.

If something does not behave as expected after handover, our engineers can help with drivers, containers and frameworks as part of expert onboarding.