How to Deploy Gemma 3 on a Dedicated GPU Server with vLLM and Docker

Learn how to deploy Google’s Gemma 3 on a dedicated Ubuntu GPU server with vLLM, Docker and systemd. Step-by-step guide.
 

Running a large language model on your own server gives you full control over your data, your latency and your costs. Google’s Gemma 3 is a strong open model family for this, and vLLM makes serving it simple.

What you will build

A Gemma 3 endpoint on a single Ubuntu GPU server, bound to localhost, managed by systemd and tested with both text and image prompts. vLLM exposes an OpenAI-compatible API, and Docker keeps the CUDA and Python dependencies isolated from the host.

Choosing a model size

Gemma 3 comes in 1B, 4B, 12B and 27B sizes. The 4B, 12B and 27B models accept images as well as text and support a 128K-token context window. As a rough guide:

• 4B: one 16 GB GPU
• 12B: one 40–48 GB GPU, or two 24 GB GPUs
• 27B: one 80 GB GPU, or multiple GPUs

bfloat16 needs an NVIDIA GPU with compute capability 8.0 or higher (Ampere or newer).

The steps in short

1. Install the NVIDIA driver on the host.
2. Install Docker and the NVIDIA Container Toolkit.
3. Store your Hugging Face token in a root-only environment file.
4. Run vLLM in a test container and confirm the model loads.
5. Test the API with text and image requests.
6. Move it into a systemd service so it starts on boot and restarts on failure.
7. Scale with tensor parallelism, add a reverse proxy for HTTPS and API keys, and monitor with Prometheus metrics.

The full guide also includes a troubleshooting table for the errors people hit most often, like 401 downloads, CUDA out-of-memory and missing GPU drivers in Docker.

Read more: https://www.fitservers.com/tutorials/howto/deploy-gemma-3-gpu-server/