Self-Hosted AI Agents: What Stays on Your Machine

A self-hosted AI agent is an agent whose runtime you install on hardware you control. In most setups that covers only one of three layers: the runtime is yours, the model is still a cloud API, and the tools reach wherever you point them. Keeping the model local is a separate decision, and it costs memory, roughly 16 GB for the smallest capable models, plus some tool-calling reliability.

What follows are field notes, not a ranking: one observation per note, with its source, checked on 2026-09-11. The check in the first section is the part to keep; the notes are the evidence.

The three-layer check before you install anything

Ask three questions about any setup, including the one you already run.

Where does the runtime run? The runtime is the process that holds your credentials and calls your tools: OpenClaw's gateway, Hermes Agent, n8n, OpenHands. It is light: a ranking page for this query says "agent orchestration logic runs on standard CPUs", and OpenClaw's own VPS page only worries about low-power VMs feeling slow, which puts this layer within reach of a $12 per month virtual server.

Where does the model run? Read every model line in your config, auxiliary models included. If it names anthropic, openai, openrouter, or an Ollama tag ending in -cloud, every prompt, customer email included, leaves the machine. The model is local only when Ollama, LM Studio, vLLM or llama.cpp serves it on hardware you own or rent.

Where do the tools reach? A fully local model can still send email, browse and push code. That is the layer where damage happens, and the reason to keep one irreversible action behind one exit door, whichever runtime you pick.

The test for the second question takes a minute. Load the model, disconnect the network, and run the agent on input you saved earlier. If it still answers, the model is local. Then run ollama ps: Ollama's FAQ explains that the Processor column reads "100% GPU", "100% CPU" or a split, which tells you whether the GPU answered. It is the same question as whether you can run the code elsewhere, asked of the model instead of the app.

A made-up case: a solo founder wants an agent that reads support mail each morning and drafts replies. In the hybrid setup, the runtime lives on a 2 GB server, the model is a cloud API, and each customer email goes to that provider; the credentials stay on the server. In the fully local setup, the same runtime needs a machine with at least 16 GB of memory for the smallest model in the notes below; the 2 GB server no longer does the job. Either way the agent gets its own mailbox, which is what AgentMail provides, rather than the founder's inbox. The verification is the network test above, run on five saved emails with sending disabled.

“

Self-hosting the runtime keeps your credentials at home; only self-hosting the model keeps your prompts there.

”

Notes on the model layer

OpenAI's gpt-oss on the Ollama library. The 20b variant is a 14 GB download and runs on systems with "as little as 16GB memory"; the 120b variant is 65 GB and fits "on a single 80GB GPU". Both are Apache 2.0 with a 128K context. The download size is a fair first estimate of the memory you need, before context.

Qwen3 on the same library. Eight sizes, from 523 MB at 0.6b to 5.2 GB at 8b, 20 GB at 32b and 142 GB at 235b. By the download-size rule above, the 8b fits a 16 GB laptop and the 32b wants a 24 GB GPU or a Mac with 32 GB or more.

OpenHands's local model page. The project recommends Qwen3.6-35B-A3B as the first local model to try, with "a recent GPU with at least 24GB of VRAM for quantized variants" or "a Mac with Apple Silicon with at least 64GB of unified memory". The same page warns: "If OpenHands behaves like a plain chatbot, refuses to use tools or files, or has constant failed tool calls with a local model, the issue may be with the model itself rather than your setup." That is the honest summary of local agents in 2026: the runtime works, the model may not call tools.

OpenClaw's Ollama page. It supports three modes, cloud plus local, cloud only, and local only, and carries one warning: do not point it at the /v1 OpenAI-compatible URL, because "it breaks tool calling and models can emit raw tool-call JSON as plain text". Use the native base URL without the suffix. The docs home adds: "For best quality and security, use the strongest latest-generation model available", which in practice points at API models.

Ollama's cloud models. Since Ollama added cloud models, ollama run gpt-oss:120b-cloud works on a laptop without a GPU because the request is "automatically offloaded to Ollama's cloud service". The command that used to mean local now means either; the suffix is the whole difference.

Hermes Agent's configuration guide. Any OpenAI-compatible endpoint works as a custom provider, Ollama included. The same page raises socket timeouts to 30 minutes for local endpoints because "local LLMs can take minutes for prefill on large contexts before producing the first token". Local is private; it is not fast on a long conversation.

RunPod's price list, updated 2026-07-27. An RTX 4090 costs $0.34 per hour on the community tier and $0.74 on the secure tier. A 730-hour month on the 4090 is $248 on community and $540 on secure. For a solo builder, a rented GPU that never sleeps is a control decision before it is a savings plan.

Notes on the runtime layer

OpenClaw's install page. Node 24.16 or 26.1 or newer, an installer script, Docker, and a long list of hosts from Hetzner to a Raspberry Pi. The VPS page's first rule is to keep the gateway on loopback and reach it through an SSH tunnel or Tailscale; binding it to a network requires a shared secret. What a gateway login grants is the subject of the security boundary between Hermes and OpenClaw, and it decides more than the feature list.

Ollama's FAQ. Ollama binds 127.0.0.1 on port 11434 by default, and OLLAMA_HOST changes that. The FAQ explains how to expose it; it does not add a login. Anything that can reach the port can run your model, so a model server on a shared network needs a proxy in front of it.

n8n's self-hosted AI starter kit. The Docker Compose template bundles n8n, Ollama, Qdrant and PostgreSQL, with profiles for CPU, Nvidia and AMD. On Apple Silicon "you can't expose your GPU to the Docker instance", so you run Ollama on the Mac and point the kit at host.docker.internal:11434. The README calls the kit "not fully optimized for production environments": a proof-of-concept box, and the fastest way to see all three layers on one screen.

If your agent answers questions from documents, test the retrieval step: the D-RAG paper analysis and retrieval evaluation checklist shows how to verify its answers.

DigitalOcean's droplet prices. The basic tier is $4 per month for 512 MiB, $12 for 2 GiB and $24 for 4 GiB. The $12 box runs the runtime layer of the fictional support agent above; none of the three has the memory for the 14 GB model.

PostHog's self-hosting page. An open source company that offers both tells you what self-hosting costs: PostHog writes that its cloud "is far and away the best experience for the vast majority of our users", offers no support for self-hosted instances, keeps paid features cloud-only, and sets the floor at a 4 vCPU, 16 GB machine. Read any agent vendor's "$0" claim with that page open.

Notes on what the ranking pages leave out

The first result is a launch post from March 2025. The developer of Observer AI, a screen-watching agent built on Ollama, posted it, and one of the first replies asks for "a couple of ELI5 examples of what agents are commonly used for". Further down, a reader explains why local is the only option for old backups holding documents they are "not legally allowed to share". That is the clean case for the fully local setup; most other cases are a preference.

A guide dated 2026-03-20 still installs the old package. It says Node 18 and npx clawdbot@latest; the project now goes by OpenClaw and its install page says Node 24.16 or 26.1. The guide's cost table lists a year of self-hosting at "$0 (local)" for an individual, with no line for the machine. Check any install command against the project's own docs before you paste it.

Decide the layer first. If the point is control over credentials and an agent that runs while your laptop sleeps, host the runtime on the $12 box and pay a cloud model per token. If the point is that the data cannot leave, buy or rent the memory, pick the model from the pages above, and run the network test before you trust the word local in anyone's title.