Self-Hosting Llama 3 and Mistral LLMs on Domain India VPS (Ollama + vLLM)

Running an open-weight language model (Llama, Mistral, Qwen, Gemma) on your own server keeps prompts and documents on hardware you control and replaces per-token API bills with a fixed monthly cost. This guide covers sizing, Ollama for the quickest setup, llama.cpp for the best CPU performance, vLLM for GPU throughput, and how to put the model safely behind an API.

Why self-host an LLM

Hosted APIs (OpenAI, Anthropic Claude, Google Gemini) are fast and very capable, but a self-hosted model makes sense when:

• Privacy — customer data, medical records or contracts must not leave your server
• Predictable cost — heavy, steady usage costs a fixed server fee instead of a per-token bill
• Customisation — you want to fine-tune on your own vocabulary or documents
• Offline or sovereign use — no dependency on a foreign API being reachable

Trade-offs:

• Quality gap — small open models are well behind the largest hosted models on hard reasoning and coding
• Ops burden — you handle updates, downtime and scaling
• Speed on CPU — expect a few tokens per second on a 7B–8B model, not the near-instant replies of a hosted API

What fits in how much RAM

Rule of thumb: a 4-bit quantised model needs roughly (parameters in billions × 0.6) GB of RAM, plus room for the context and the operating system. An 8B model needs about 5 GB; a 70B model needs 40 GB or more.

Speeds depend on the CPU generation, the number of cores and the quantisation, so benchmark on your own server before you promise response times.

Option A — Ollama (easiest)

Ollama bundles model download, serving and a REST API in one program.

1. SSH into your server as root (or a sudo user).
2. Install:

   curl -fsSL https://ollama.com/install.sh | sh

3. Download a model (sizes are approximate for the default 4-bit builds):

   ollama pull llama3.2:3b     # ~2 GB, fast
   ollama pull mistral:7b      # ~4 GB, balanced
   ollama pull qwen3:8b        # ~5 GB, strong multilingual

4. Test:

   ollama run mistral:7b "Explain how a TCP handshake works in 3 sentences."

5. Call the REST API (Ollama listens on 127.0.0.1:11434 by default):

   curl http://127.0.0.1:11434/api/generate -d '{
     "model": "mistral:7b",
     "prompt": "Summarise: ...",
     "stream": false
   }'

Ollama also offers an OpenAI-compatible endpoint at /v1/chat/completions, so most OpenAI SDKs work by changing the base URL.

systemd + nginx front

On Linux the install script creates and starts an ollama systemd service. To keep a model loaded instead of unloading it after 5 idle minutes, set OLLAMA_KEEP_ALIVE:

sudo systemctl edit ollama
# add:
# [Service]
# Environment="OLLAMA_KEEP_ALIVE=30m"
sudo systemctl restart ollama

To reach it from outside, leave Ollama on localhost and publish it through nginx with TLS and authentication:

server {
    listen 443 ssl;
    server_name llm.yourcompany.com;
    # ssl_certificate / ssl_certificate_key: issue with certbot (Let's Encrypt)

    location / {
        auth_basic "LLM";
        auth_basic_user_file /etc/nginx/.htpasswd-llm;
        proxy_pass http://127.0.0.1:11434;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;          # stream tokens as they arrive
        proxy_read_timeout 300s;
    }
}

Option B — llama.cpp (best CPU performance)

Ollama is built on llama.cpp. Use llama.cpp directly when you want full control of threads, context and quantisation.

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build --config Release -j "$(nproc)"

# Download a GGUF model file (Q4_K_M is a good default) from Hugging Face,
# then serve it on localhost only:
./build/bin/llama-server -m models/your-model-Q4_K_M.gguf \
    -c 4096 --host 127.0.0.1 --port 8080

llama-server also exposes an OpenAI-compatible API. Put it behind nginx with authentication, exactly as for Ollama.

Option C — vLLM (GPU throughput)

vLLM serves many concurrent requests efficiently with continuous batching. It is built for GPUs; its CPU backend exists but is rarely the right choice, so on a CPU-only server prefer Ollama or llama.cpp.

python3 -m venv ~/vllm && source ~/vllm/bin/activate
pip install vllm

# OpenAI-compatible server
vllm serve meta-llama/Llama-3.2-3B-Instruct \
    --host 127.0.0.1 --port 8000 \
    --max-model-len 8192 \
    --dtype auto

Meta's Llama models on Hugging Face are gated: accept the licence on the model page and log in with huggingface-cli login first.

Then call it with any OpenAI SDK:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="meta-llama/Llama-3.2-3B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

The rest of your app does not need to know the model is self-hosted.

Picking a model

Open models change quickly; check each model's licence and the current release before you commit.

For Indian languages, test multilingual families such as Qwen3 and Gemma 3, and models built in India for Indic languages, for example from Sarvam AI. Always evaluate on your own sample prompts in Hindi, Tamil or whichever language you need.

Production patterns

Pattern 1 — API key gateway

Don't expose the model server directly. Put a small API in front that validates keys, rate-limits per key, logs usage and can fall back to a paid API.

# FastAPI gateway example (pip install fastapi uvicorn httpx)
import os, time
from collections import defaultdict
from fastapi import FastAPI, HTTPException, Depends, Header
import httpx

app = FastAPI()
VALID_KEYS = set(os.environ["GATEWAY_KEYS"].split(","))   # comma-separated
hits = defaultdict(int)   # in-memory; use a database for several processes

async def validate_key(x_api_key: str = Header()):
    if x_api_key not in VALID_KEYS:
        raise HTTPException(401, "Invalid key")
    return x_api_key

@app.post("/v1/chat/completions")
async def chat(req: dict, api_key: str = Depends(validate_key)):
    minute = int(time.time() // 60)
    hits[(api_key, minute)] += 1
    if hits[(api_key, minute)] > 60:
        raise HTTPException(429, "Rate limit")

    async with httpx.AsyncClient(timeout=300) as client:
        resp = await client.post("http://127.0.0.1:11434/v1/chat/completions", json=req)
    return resp.json()

Pattern 2 — Keep the model warm

Loading a model into RAM takes seconds to tens of seconds. Rather than pinging it from cron, set OLLAMA_KEEP_ALIVE (above), or pass "keep_alive": "30m" in API requests. Leave enough free RAM for the model to stay resident.

Pattern 3 — Fallback chain

try:
    return await local_chat(prompt)     # self-hosted
except (TimeoutError, httpx.HTTPError):
    return await hosted_chat(prompt)    # paid fallback

You pay the API only during peaks or outages. Tell users if their data may go to a third-party API in that case.

Common pitfalls

Running this on Domain India

LLM serving needs root access, several GB of RAM and a long-running process, so it belongs on a VPS, not shared hosting or the App Platform.

• Domain India VPS plans are self-managed KVM servers with full root access and NVMe storage. The catalogue lists RAM from 2 GB (VPS Starter) up to 32 GB (VPS Enterprise); no GPU is listed, so plan for CPU inference with the model sizes in the table above.
• You install and update the software yourself. No backups or snapshots are included with a VPS, so keep copies of your configuration and data elsewhere.
• Shared hosting (cPanel, DirectAdmin, Webuzo) stops long-running processes and cannot run a model server. You can still call a model on your VPS, or a hosted API, from a PHP or Node.js site on shared hosting over HTTPS.

FAQ

Ready to run your own model? Compare VPS plans, or open a support ticket if you are unsure which size fits.