Local Llama Server

Apis AI can run local .gguf models through a llama.cpp server. The model you pick in the dropdown is really loaded, the server starts and stops with the app (and after an idle timeout), and you can give every model its own server arguments (flags).

On This Page

Setup
Model Selection
Start, Stop & Model Change
Idle Stop & Manual Stop
Visible Startup Status
Server Arguments
Presets
Example: 35B MoE
Prefill & Restart Speed
Context Window & Tool Calls
Troubleshooting

Setup

Install llama.cpp and make sure llama-server (or llama) is on your PATH or in ~/.local/bin.
Add a provider whose base URL points at the local server, e.g. http://localhost:8080/v1 (the predefined Llama (lokal) provider already does this).
Under Advanced settings add one or more folders with .gguf models (one per line).
Pick a model in the model dropdown and, optionally, enable Start llama server automatically.

Model Selection

The dropdown lists the model files found in your configured folders, shown as the file name without the .gguf extension. When you send a request, Apis AI maps that name back to the actual file and starts the server with exactly that model.

If the model field contains a full path to a .gguf file, that path always wins.
Otherwise the scanned files are matched by file name. Files are sorted deterministically, so the result is reproducible.
If the selected name cannot be resolved, Apis AI shows a warning in the chat and logs it – instead of silently loading a different model.

Tip If two folders contain files with the same name, use the full path in the model field to pick a specific one unambiguously.

Start, Stop & Model Change

Apis AI only starts a server when nothing answers on the configured URL yet. A server you started yourself (or that was already running) is adopted and never touched.

The app stops the server it started itself when it closes, so a local model does not keep GPU memory occupied afterwards.
After an idle timeout without a real request the app also stops the server and frees the VRAM (see below); the next request starts it again automatically.
When you change the model while the app is running, the next request stops the old server and restarts it with the new model.
If a server is adopted from outside, model and flag changes take effect only after that server is restarted – the settings show a hint.
Automatic start (on app start and before a request) only happens when Start llama server automatically is enabled. When it is off, start your own server; Apis AI connects to it.
The loopback host is normalized to 127.0.0.1, so Apis AI and a hand-started server on 127.0.0.1 see each other instead of running in parallel (IPv4 vs. IPv6).
A large model can take minutes to load. Apis AI waits for readiness and reports progress in the server log; the model list itself loads immediately from your folders.
On an orderly stop the app saves the loaded context to disk (--slot-save-path) and restores it when the same model starts again - so the next request does not re-evaluate the whole conversation.

Idle Stop & Manual Stop

A server started by Apis AI no longer keeps the GPU memory occupied forever. Under Advanced settings the field Server stop after inactivity (minutes) controls how long the server may stay idle without a real chat request:

Default: 5 minutes. After that time without a request the app stops the server and frees the VRAM; the app log records the stop with the PID and the idle time, and the chat shows a short notice.
0 = off: the server keeps running until the app is closed.
Only real model requests count as activity. The periodic model-list refresh and reachability/health checks do not keep the server alive.
A running generation is never interrupted; the timer only runs while nothing is in flight.
The next request starts the server again automatically – the model then has to be loaded again, which can take a while (the waiting bubble shows Starte Server… / Lade Modell…).
The setting takes effect immediately, without restarting the app.

Next to it there is a manual Stop local server button (mirroring the Whisper server button). It stops only a server that Apis AI started itself; a server you started manually is never touched. The line below the button shows whether the app-managed server is currently running and, if so, which model it loaded.

Visible Startup Status

While you wait for the first answer, the chat shows the real progress instead of a single generic message. For the local Llama provider the waiting bubble steps through three stages:

Starte Server… – the llama-server process has been launched and the port is not answering yet.
Lade Modell… – the port answers, but llama.cpp is still loading the model (HTTP 503 “Loading model”).
Verarbeite Prompt… – the model is loaded and the server evaluates the prompt (prefill) before the first token. For long contexts this can take many seconds; when the server log is readable, the app additionally shows the prefill progress (e.g. 64 %) parsed from it.

If the server is already warm, the bubble goes straight from the generic Antwort wird vorbereitet… to Verarbeite Prompt…. Other providers (OpenAI, Anthropic, Ollama, DeepSeek, …) keep the previous, unchanged text. The stages only appear in the waiting bubble and never end up as a message in the chat history; every stage change is also written to the app log.

Loopback only Apis AI only auto-starts the server on a loopback address (127.0.0.1). A llama.cpp server has no authentication, so it is never bound to a network address (0.0.0.0 or a LAN IP). For LAN access, start the server yourself and point the Base URL at it.

Server Arguments

Under Advanced settings → Server arguments you can add extra command-line flags per provider. Each row has an enable checkbox, the flag name (with autocomplete), and an optional value. Switching a flag off keeps the row (greyed out) so you can re-enable it without retyping.

Element	Meaning
Scope	All models or a single model. Model-specific flags override equally named general flags.
Flag	Without leading dashes (e.g. `n-cpu-moe`, `fa`, `ngl`). The exact spelling comes from the `llama-server --help` catalog: short aliases keep a single dash (`-c`, `-fa`, `-ngl`), long forms two (`--jinja`, `--flash-attn`). Names outside the catalog fall back to the length rule (one dash for a single character, two otherwise).
Value	Empty for pure switch flags such as `jinja` or `no-reasoning-preserve`. Value flags are sent as two tokens (`-ngl 99`), which is the form `llama.cpp` expects. A known value flag without a value is skipped (and marked) so it cannot swallow the next flag.
Presets	Set several flags at once and can be toggled on and off.
Command preview	Read-only line showing exactly what would be started (paths with spaces are quoted).

Apis AI always appends the following managed arguments last; they cannot be overridden: -m / --model, --host and --port. The Context window from the advanced settings is applied as -c automatically; an explicit c/ctx-size argument overrides it.

Unknown Flags Flags not found in the llama-server --help catalog are allowed but marked as a warning, so a newer llama.cpp build with new flags still works.

Presets

Presets bundle common configurations into one click:

Preset	Sets
GPU full	`ngl=99`
MoE: experts in RAM	`ngl=99`, `n-cpu-moe=40`
CPU-only	`ngl=0`
Context 32k	`c=32768`
Flash Attention	`fa=on`, sent as `-fa on` (llama.cpp requires an explicit value; `on\|off\|auto`)
Jinja template (tool calls)	`jinja`
Reasoning off	`reasoning-budget=0` (keeps reasoning from eating the output budget)
Threads 8	`threads=8`
Parallel 1	`parallel=1` (a single context slot)

What is the Jinja template? A .gguf model ships its own chat template: the rules that turn messages into the exact prompt the model was trained on. --jinja makes llama-server use that template's Jinja2 form instead of a generic built-in one. Many models (Qwen, Hermes, …) define their tool-calling format inside this template, so tool calls often only work with --jinja enabled. If a model has no Jinja template, the flag is harmless.

Example: 35B MoE

A large Mixture-of-Experts model that does not fit into VRAM can be split across GPU and RAM:

ngl             = 99
n-cpu-moe       = 40
c               = 32768
fa              = on
jinja           = (empty)
reasoning-budget = 0

The resulting start command looks like this:

llama-server -ngl 99 --n-cpu-moe 40 -c 32768 -fa on --jinja --reasoning-budget 0 \
  -m /mnt/models/Hermes3.6-35B-A3B.gguf --host 127.0.0.1 --port 8080

Flash Attention & short aliases In current llama.cpp builds -fa takes an optional value (on|off|auto). Sent without a value it consumes the next argument and the server refuses to start, so Apis AI always sends value flags as two tokens (-fa on). Short aliases such as -fa and -ngl take exactly one dash - the double-dash forms --fa/--ngl are rejected ("invalid argument") and would abort the server start, which then shows up as a connection error in chat.

Prefill & Restart Speed

Before the first token appears, llama.cpp has to process the whole prompt (system prompt, tool definitions and the conversation). This is the prefill - pure computation, so its duration is prompt tokens ÷ prefill throughput. In the waiting bubble this is the Verarbeite Prompt… stage; for a full 32k context it can take well over a minute on the first request.

Apis AI keeps that warm-up cheap in two ways:

KV cache across restarts: the app starts the managed server with --slot-save-path, saves the loaded context when it stops the server and restores it on the next start of the same model. The first request after a restart then only evaluates the new part instead of the whole conversation. The cached file lives under llama-slots/ in the config directory, is overwritten on every stop and can get large for big contexts (it holds the whole KV cache; the size depends on the model). If the context cannot be restored (different model, changed history), llama.cpp simply reuses the common prefix or re-evaluates - never a wrong context and never an error.
Progress in the waiting bubble: while the prompt is evaluated, the app reads llama.cpp's own progress lines and shows e.g. Verarbeite Prompt… 64 %. If the log is not readable, the plain text stays.

Server flags that affect prefill

These flags stay a user decision (add them under Server arguments); Apis AI never changes them on its own. The effect depends on the model, the hardware and how much RAM/VRAM you can spare.

Flag	Effect / risk
`load-mode` = `none`	Loads the model fully into memory instead of memory-mapping it. Removes the slow first pass caused by a cold page cache (the server logs a hint about this). Needs enough RAM/VRAM for the whole file.
`batch-size`, `ubatch-size`	Larger batches process more tokens per step and can raise prefill throughput, at the cost of more memory.
`cache-reuse`	Minimum chunk size for reusing KV parts via shifting - helps when the prompt shares a long prefix with the previous one.
`parallel` = `1`	A single context slot. Removes the throughput loss when several requests run at once; recommended when you only chat with one model at a time.

Fewer prompt tokens = faster prefill Roughly every saved 1,000 prompt tokens is 4-6 seconds less prefill. The two biggest levers are the tool definitions (cut tools per model under Settings → Tool Selection per Model) and a lower compression threshold (compress earlier = fewer tokens, but less original context). Both are your call and are not changed automatically.

Two models on one URL One llama.cpp server serves one model at a time. Two chats that use different local models on the same URL each force a restart when you switch between them. The saved KV cache above avoids the expensive re-prefill, but the model still has to be loaded. For working with several models at once, start one server per model on its own port and add a provider for each.

Context Window & Tool Calls

The Context window value in the advanced settings is passed to the started server as -c, so the setting and the server always agree. Tool calls need room in that window: every enabled MCP server and tool group adds tool definitions to the prompt. With many tools a 32k window can be too small even for the first message.

If the first message does not fit, Apis AI tries to read the real context size from the server (/props) first. If it still does not fit, the tools are disabled for that message and the chat shows a clear notice. If the prompt later exceeds the server window, the error names the prompt and context token counts and suggests raising the window or reducing the tools.

Tip For large tool sets either raise the context window (e.g. 40960 or more) or cut tools per model under Settings → Tool Selection per Model. Start a new chat afterwards so the tools become active again.

Troubleshooting

Symptom	Cause / Fix
Warning "model ... not found" in chat	The selected name does not match any `.gguf` file. Check the folders or enter the full path in the model field.
Connection refused / "Verbindungsaufbau abgelehnt" when starting the local server	The server process exited before binding the port - usually an invalid server argument. Apis AI then shows the concrete cause in chat: the `error:` line from the log (e.g. `invalid argument: --fa`) plus the start command, instead of a bare connection error. Short aliases need one dash (`-fa`, `-ngl`), long forms two (`--flash-attn`, `--n-gpu-layers`). Full log: `logs/llama-server.log`.
Changes to model/flags have no effect	An externally started server answers on the URL and is adopted. Restart it, or let Apis AI manage it.
Model still loading (HTTP 503)	Apis AI waits and retries automatically until the model is ready; the waiting bubble shows Lade Modell… during that time.
GPU memory stays occupied	Check whether a local Llama server is still active. Apis AI stops only the server it started, after the configured idle time (default 5 min) or via the Stop local server button; a server you started yourself must be stopped by you.
First request is very slow ("Verarbeite Prompt…")	That is the prefill of the whole prompt. On a restart the saved KV cache is restored automatically; see Prefill & Restart Speed for the flags and levers that reduce it.
HTTP 500 "System message must be at the beginning"	Some chat templates only allow a `system` message as the very first message. Apis AI normally sends the `system` role for every system message; only when a model rejects the request with this error does it switch to the fallback and wrap later system messages (reminders, triggered rules, project instructions) as user messages - and remembers that per model.