Private network between GPU servers

A private network between GPU servers lets your machines talk to each other on addresses the internet cannot reach. At OverGPU it is available on request: our team links the servers you rent on an isolated network, and you use it for data pipelines, multi-node jobs and every service that has no reason to sit on a public IP.

Why use a private network between GPU servers

Every OverGPU server has a public IPv4 address and an IPv6 /64, with DDoS mitigation on its public IPs. That is what your users and your SSH sessions need. Traffic between your own servers is a different matter:

• Exposure. A vector database, a Redis queue or a metrics endpoint on a public IP is one firewall mistake away from the internet. On the private network, it cannot be reached from outside.
• Simpler rules. Internal services listen on private addresses only, so the public firewall stays short: SSH and the ports your users need.
• Separation. Checkpoint copies and dataset transfers between your servers stay off the public interface that serves your users.

You may not need one. A single server, or servers that only talk to the outside world, gain nothing from a private network. It starts to matter as soon as two of your machines exchange data or depend on each other's services.

Common setups

Training node feeding serving servers

An HGX node trains or fine-tunes, and single-GPU servers serve the result. Checkpoints move to the serving machines over the private network, for example with rsync, and only the inference endpoints face the public internet.

Data pipelines

One server preprocesses, tokenizes or embeds data while others train or index it. Shards, embeddings and logs travel between them on private addresses.

Multi-node jobs

Distributed evaluation, hyperparameter sweeps, Ray workers and data-parallel jobs that exchange modest amounts of data fit a private network well. Tightly coupled training at scale is a different case, covered in the next section.

Monitoring and internal APIs

Metrics exporters, log shippers and internal model APIs listen on the private side only. Our GPU monitoring guide covers the metrics worth collecting.

Private network or cluster fabric?

A private network is not the same as the high-speed fabric of a GPU cluster. Training runs that synchronize gradients across nodes on every step need RDMA over InfiniBand or RoCE. When one such job needs more GPUs than an 8-GPU server holds, a GB300 NVL72 rack puts 72 GPUs in a single NVLink domain. The private network carries everything around that: data movement, orchestration, services and lighter distributed work.

Throughput depends on the servers you link. Each server's network specification is listed in our catalog, for example 25 Gbps on an H100 single-GPU plan and 10 Gbps on an L40S plan. Describe what you plan to move between servers in your order notes or a support ticket, so our engineers can plan for it.

Security on the private side

• Bind internal services to the private interface, not to 0.0.0.0.
• Keep a host firewall on the public interface: allow SSH and the ports your users need, and deny the rest. The networking and firewall guide shows a baseline.
• Keep authentication on internal services. Isolation limits who can reach them; it does not replace credentials.
• Docker's published ports can bypass host firewall rules. Publish internal containers on the private address only, for example -p <private-ip>:6379:6379.

DDoS mitigation protects your public IPs, while the private network is not reachable from the internet in the first place. Our DDoS protection page covers the public side.

Using the private network day to day

Once it is set up, run ip -br addr on each server to see the private interface and its address. Then give your servers stable names, so scripts and configs never depend on public IPs. For example, in /etc/hosts on each server:

<private-ip-1>  train-1
<private-ip-2>  serve-1

From there, internal traffic uses those names. A checkpoint copy from the training node to a serving node becomes:

rsync -a --info=progress2 /data/ckpt/ serve-1:/data/ckpt/

Point internal services at the same names: a database or vector store listening on the private address, a metrics scraper reading exporters on the private side, or an internal model API that only your other servers call. Keep the names identical on every server, so a node can be replaced without editing every config.

How to request a private network

1. Mention it in the order notes when you deploy your servers, or open a support ticket for servers you already rent.
2. List the servers to link and what will travel between them.
3. Our team sets it up and sends you the details for each server.

It is easiest to plan when you mention it with your first order, especially for several 8-GPU servers.