# Serve an LLM with vLLM

> Serve an LLM with vLLM on your GPU server. Set up Python with uv, install PyTorch and vLLM, test the OpenAI-compatible API and run it as a service.

Source: https://overgpu.com/docs/pytorch-vllm  
Updated: 2026-10-22  
Publisher: OverGPU (https://overgpu.com/)

This guide shows how to serve an LLM with vLLM on an OverGPU server: create a Python environment, install PyTorch and vLLM, start an OpenAI-compatible endpoint on one GPU or across eight, test it with curl and run it as a systemd service. Before you pick a model, estimate its memory with the [VRAM calculator](https://overgpu.com/tools/vram-calculator). In BF16, weights take roughly 2 bytes per parameter, plus room for the KV cache.

## Install PyTorch and vLLM in a Python environment

### Create the environment

uv creates environments and installs packages quickly, and can fetch a Python version your OS does not ship:

```
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv ~/venvs/vllm --python 3.12
source ~/venvs/vllm/bin/activate
```

With the standard tools instead, run `sudo apt install -y python3-venv`, then `python3 -m venv ~/venvs/vllm` and the same `source` line. Keep vLLM in its own environment, separate from your training code.

### Install the packages

```
uv pip install vllm --torch-backend=auto
python -c "import torch, vllm; print(torch.__version__,
  torch.cuda.is_available(), vllm.__version__)"
```

You should see `True` between the two versions. vLLM pins the PyTorch version it was built against and installs it for you, and `--torch-backend=auto` lets uv pick the PyTorch build that matches your driver. Do not install another PyTorch version on top. For a training environment without vLLM, install PyTorch alone the same way with `uv pip install torch --torch-backend=auto`. With plain pip, run `pip install vllm`.

## Start the OpenAI-compatible server

On a single GPU, a 7B-class model is a good first test:

```
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90
```

On an 8-GPU server, split a larger model across all GPUs with tensor parallelism:

```
vllm serve Qwen/Qwen2.5-72B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --tensor-parallel-size 8 \
  --max-model-len 32768
```

The first start downloads the weights, which can take a while. The server is ready when the log shows `Application startup complete`. Weights go to `~/.cache/huggingface`; to keep them on your [NVMe data drive](https://overgpu.com/docs/storage-nvme), run `export HF_HOME=/data/hf` before starting.

| Option | What it does |
| --- | --- |
| `--tensor-parallel-size` | Number of GPUs that share the model. It must divide the model's attention head count; 8 works for most large models. |
| `--max-model-len` | Longest context (prompt plus output) per request. Lower it if vLLM reports that the KV cache cannot hold the maximum length. |
| `--gpu-memory-utilization` | Share of each GPU's memory vLLM may use for weights and KV cache, 0.9 by default. Lower it if other processes use the GPU. |
| `--host` | Address to listen on. Without it vLLM listens on all interfaces, so keep `127.0.0.1`. |

## Test the API with curl

```
curl http://127.0.0.1:8000/v1/models

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-7B-Instruct",
       "messages": [{"role": "user", "content": "Say hello in five words."}],
       "max_tokens": 32}'
```

The first call should list the model. The second returns JSON with the answer in `choices[0].message.content`. Any OpenAI-compatible client works with its base URL set to `http://127.0.0.1:8000/v1`. From your laptop, reach it through an SSH tunnel: `ssh -N -L 8000:127.0.0.1:8000 gpu1`.

## Use a Hugging Face token for gated models

Some models, such as Meta's Llama family, require you to accept a license on Hugging Face. Accept it on the model page with your account, create an access token with read permission, then pass it to vLLM in the `HF_TOKEN` variable. Typing it at a prompt keeps it out of your shell history:

```
read -rsp 'Hugging Face token: ' HF_TOKEN && export HF_TOKEN
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 127.0.0.1 --port 8000 --max-model-len 16384
```

A 401 or 403 error during the download means the token is missing or wrong, or the license was not accepted for that account.

## Run vLLM as a systemd service

1. Store settings and secrets in a file only root can read. Generate the API key with `openssl rand -hex 32`; vLLM reads `VLLM_API_KEY` as if you had passed `--api-key`. `sudo mkdir -p /etc/vllm sudo tee /etc/vllm/vllm.env >/dev/null <<'EOF' HF_HOME=/data/hf HF_TOKEN=hf_your_token_here VLLM_API_KEY=replace-with-a-long-random-string EOF sudo chmod 600 /etc/vllm/vllm.env`
2. Create `/etc/systemd/system/vllm.service`. This example runs as root with the environment created above; adjust `User` and the paths for another account: `[Unit] Description=vLLM OpenAI-compatible server After=network-online.target Wants=network-online.target [Service] User=root EnvironmentFile=/etc/vllm/vllm.env ExecStart=/root/venvs/vllm/bin/vllm serve Qwen/Qwen2.5-7B-Instruct \ --host 127.0.0.1 --port 8000 --max-model-len 16384 Restart=on-failure RestartSec=10 [Install] WantedBy=multi-user.target`
3. Start it and follow the log: `sudo systemctl daemon-reload sudo systemctl enable --now vllm sudo journalctl -u vllm -f` When the log shows `Application startup complete`, the curl tests work again once you add the header `-H "Authorization: Bearer YOUR_KEY"`. Without it, you should get 401 Unauthorized. The service restarts after a crash and starts at boot.

## Alternative: the vllm/vllm-openai image

```
docker run -d --name vllm --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v /data/hf:/data/hf \
  --env-file /etc/vllm/vllm.env \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-7B-Instruct --max-model-len 16384
```

The image's entrypoint is the vLLM server, so arguments after the image name go straight to it. The env file sets `HF_HOME=/data/hf`, which is why that path is mounted. Pin a version tag instead of `latest` in production. On AMD servers, use the ROCm image from [ROCm setup](https://overgpu.com/docs/amd-rocm).

## Secure the endpoint

For personal use, keep vLLM on `127.0.0.1` and use the SSH tunnel. To serve applications over the internet, put a reverse proxy with TLS in front, require an API key and expose only the `/v1/` paths: the key protects the `/v1` routes, and other routes such as `/metrics` may stay open. The [firewall guide](https://overgpu.com/docs/networking-firewall) covers ufw, Docker's published ports and a Caddy or nginx setup. For sizing and running costs, read [self-hosting LLM inference with vLLM](https://overgpu.com/blog/self-host-llm-inference-vllm).

## Troubleshooting

- **CUDA out of memory at startup.** Lower `--max-model-len`, spread the model over more GPUs with `--tensor-parallel-size`, or choose a smaller or quantized model. Check with `nvidia-smi` that no other process holds GPU memory.
- **Attention heads not divisible by the tensor parallel size.** Pick a size that divides the model's head count, such as 4 instead of 8.
- **401 Unauthorized.** The `Authorization` header is missing or the key does not match.

## Frequently asked questions

### Which GPU do I need to serve a 70B model with vLLM?

Roughly 140 GB for the weights in BF16 or 70 GB in FP8, plus room for the KV cache. One H200 serves a 70B model comfortably in FP8; in BF16 a single H200 is tight, so use a B200 or B300, or split the model over several GPUs with tensor parallelism.

### Does vLLM run on AMD Instinct MI355X servers?

Yes. Use AMD's ROCm-enabled vLLM images with the ROCm device flags, as shown in our ROCm setup guide. The vllm serve options are the same.

### How do I update vLLM?

In the same environment, run uv pip install -U vllm --torch-backend=auto, then restart the service. Test new versions in a separate environment first if the endpoint serves production traffic.

## Related pages

- [How to self-host LLM inference with vLLM](https://overgpu.com/blog/self-host-llm-inference-vllm): Size the GPU, run an OpenAI-compatible server, tune FP8, prefix caching and parallelism, and secure it for production.
- [GPU servers for LLM inference, running 24/7 at a flat price](https://overgpu.com/solutions/inference): KV cache sizing, FP8 and FP4, and 20–50 TB of bandwidth included. See the sizing math.
- [How much VRAM do I need for an LLM?](https://overgpu.com/blog/vram-for-llms): Quick answers for 7B, 14B, 32B, 70B and 405B models at BF16, FP8 and 4-bit, plus KV cache by context and the GPU that fits.
- [Docker GPU containers with the NVIDIA Container Toolkit](https://overgpu.com/docs/docker-containers): Use --gpus, select devices, reserve GPUs in Compose, fix PyTorch shared memory and run ROCm containers.
- [GPU server firewall setup](https://overgpu.com/docs/networking-firewall): Allow SSH first, filter IPv6, avoid the Docker port trap and secure a vLLM endpoint with TLS and an API key.