Overview
This tutorial shows how to deploy a secure, GPU-accelerated local AI stack on Ubuntu 24.04 using Docker Compose. We will run Ollama for model inference and Open WebUI as a sleek web interface, fronted by Traefik for reverse proxy, HTTPS (Let’s Encrypt), and basic authentication. You will get a production-ready setup that supports NVIDIA GPUs and is protected with TLS and a login prompt.
Prerequisites
- Ubuntu 24.04 LTS with a modern NVIDIA GPU (e.g., RTX series).
- A public DNS record (e.g., ai.example.com) pointing to your server’s IP.
- Ports 80 and 443 open in the firewall or cloud security group.
- A sudo-enabled user on the server.
- Docker and the Docker Compose plugin installed (Ubuntu’s docker.io + docker-compose-plugin or Docker’s official packages).
- An email address for Let’s Encrypt certificates.
1) Install NVIDIA Driver and Container Toolkit
First, make sure the proprietary NVIDIA driver is installed and working. On Ubuntu 24.04, the recommended driver is usually offered by “Additional Drivers” or via apt:
sudo apt update && sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers install
sudo reboot
After reboot, confirm the driver and GPU are detected:
nvidia-smi
Now install the NVIDIA Container Toolkit so Docker can access the GPU:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit.gpg
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
2) Prepare Docker Network and Directory
Create a dedicated network and a working directory for the stack:
docker network create ai_net || true
mkdir -p ~/ai-stack/{letsencrypt,ollama}
3) Create HTTP Basic Auth for Open WebUI
We will protect the web UI with Traefik’s basic auth. Generate a bcrypt hash with htpasswd. Note: when placing the hash in docker-compose labels, escape each $ as $$.
sudo apt install -y apache2-utils
htpasswd -nbB aiadmin 'StrongP@ssw0rd!'
You will get output like aiadmin:$2y$05$abc.... Copy it for the next step and remember to replace each $ with $$ in the compose file.
4) Write docker-compose.yml
Create ~/ai-stack/docker-compose.yml with the following content. Replace ai.example.com and email@domain.com with your values. Also paste your basic auth user:hash in the indicated line, with dollars escaped as $$.
version: '3.8'
services:
traefik:
image: traefik:v3.0
container_name: traefik
command:
- --api.dashboard=false
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --providers.docker=true
- --providers.docker.exposedbydefault=false
- --certificatesresolvers.le.acme.email=email@domain.com
- --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
- --certificatesresolvers.le.acme.httpchallenge=true
- --certificatesresolvers.le.acme.httpchallenge.entrypoint=web
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./letsencrypt:/letsencrypt
networks:
- ai_net
ollama:
image: ollama/ollama:latest
container_name: ollama
environment:
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
volumes:
- ./ollama:/root/.ollama
networks:
- ai_net
deploy:
resources:
reservations:
devices:
- capabilities: [gpu]
openwebui:
image: ghcr.io/open-webui/open-webui:latest
container_name: openwebui
environment:
- OLLAMA_BASE_URL=http://ollama:11434
depends_on:
- ollama
networks:
- ai_net
labels:
- traefik.enable=true
- traefik.http.routers.openwebui.rule=Host(`ai.example.com`)
- traefik.http.routers.openwebui.entrypoints=websecure
- traefik.http.routers.openwebui.tls.certresolver=le
- traefik.http.services.openwebui.loadbalancer.server.port=8080
- traefik.http.routers.openwebui.middlewares=openwebui-auth
- traefik.http.middlewares.openwebui-auth.basicauth.users=aiadmin:$$2y$$05$$REPLACE_WITH_YOUR_HASH
networks:
ai_net:
external: true
Notes: The deploy.resources.devices section in Compose is ignored outside Swarm, but many users report it still helps Compose detect GPUs with the NVIDIA Container Toolkit. If your GPU is not picked up, add runtime: nvidia under the ollama service or map NVIDIA devices explicitly.
5) Launch the Stack
Start everything in the background:
cd ~/ai-stack
docker compose up -d
Confirm that Traefik, Ollama, and Open WebUI are running:
docker ps
Wait 10–30 seconds for Let’s Encrypt to issue the certificate. Then visit https://ai.example.com. You should see a basic auth prompt. Enter your credentials and access Open WebUI.
6) Pull a Model and Test
From the Open WebUI interface, add a model like llama3:8b or phi3:mini. Alternatively, pull via CLI:
docker exec -it ollama ollama pull llama3:8b
docker exec -it ollama ollama run llama3:8b "Explain containers in one paragraph."
Open WebUI will connect to Ollama at http://ollama:11434 and use your GPU to accelerate inference if it is available.
Troubleshooting
- If nvidia-smi fails on the host, fix the driver first. The container cannot use a GPU your OS cannot see.
- If Open WebUI shows a connection error, check logs: docker logs openwebui -f and docker logs ollama -f.
- If certificates are not issued, ensure ports 80/443 are open and DNS is correct. Review docker logs traefik -f.
- If basic auth is not accepted, verify you escaped $ characters as $$ in the label.
- To keep models between upgrades, never delete the ./ollama folder.
Optional Hardening
- Restrict access by IP allowlists with Traefik middlewares in addition to basic auth.
- Set LOG_LEVEL=ERROR in Traefik if you want quieter logs.
- If exposing the Ollama API externally, add its own router, TLS, and auth. By default in this guide, the Ollama API is internal only.
Maintenance
Update images periodically and redeploy:
cd ~/ai-stack
docker compose pull
docker compose up -d
To stop the stack:
docker compose down
You now have a secure, GPU-ready, self-hosted AI chat environment with automatic HTTPS on Ubuntu 24.04, powered by Docker, Traefik, Ollama, and Open WebUI.
Comments
Post a Comment