Ollama Local LLM Server

Overview

Ollama is what I use to run LLMs locally. It manages model downloads, handles inference, and exposes a clean HTTP API that everything else on the stack talks to. Once it's running, OpenWebUI provides a browser-based chat interface, sonnet-tts handles voice output, and Open Heavens covers text-to-speech for other workflows.

Nothing leaves the homelab. No API keys per query, no usage bills, no data going to third parties.

What's Connected

• Ollama — model server, API on port 11434
• OpenWebUI — browser chat UI, connects to Ollama the same way any client would
• sonnet-tts — takes text and speaks it via a locally served TTS model, fed by Ollama
• Open Heavens — secondary TTS setup I use for specific voice workflows like listening to note summaries

Installing Ollama

curl -fsSL https://ollama.com/install.sh | sh

Ollama runs as a systemd service after install. API listens on http://localhost:11434 by default.

Pull models you want available:

ollama pull llama3
ollama pull mistral
ollama pull codellama

Quick test:

ollama run llama3

Docker Compose Setup

Running Ollama and OpenWebUI together in Compose is the cleaner approach for homelab:

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    volumes:
      - ollama-data:/root/.ollama
    ports:
      - "11434:11434"
    restart: unless-stopped

  openwebui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: openwebui
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - openwebui-data:/app/backend/data
    ports:
      - "3000:8080"
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama-data:
  openwebui-data:

OpenWebUI will be at http://your-server:3000. Sign in on first visit to create an admin account.

Text-to-Speech

sonnet-tts

sonnet-tts uses Ollama's API to pipe model output through a local TTS pipeline. Point it at the Ollama host URL and configure the voice settings in its config. Useful for getting audio out of any text generation workflow without sending audio data externally.

Open Heavens

Open Heavens is what I use for turning notes and longer content into audio. It's wired to the same Ollama instance for the language processing layer, then handles voice synthesis locally. I use it mostly for listening to summaries during commutes — saves reading a screen.

GPU Passthrough (Optional)

If you have a GPU, Ollama will use it automatically when running bare metal. For Docker, pass through the GPU:

services:
  ollama:
    image: ollama/ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Without GPU, everything runs on CPU. It's slower but completely functional for most models under 7B.

Why Local Matters

Running LLMs locally isn't just about privacy. It's about having a model that's always on, never rate-limited, and works when the internet is flaky. For a homelab that's supposed to be self-sufficient, Ollama is the core of the AI layer — OpenWebUI, TTS tools, and Hermes all depend on it.