A post titled “I built a low-latency AI companion that plays Skyrim with me” hit the Hacker News front page this week, describing a custom voice pipeline — Qwen3-ASR for speech recognition, a fine-tuned local model, a proprietary TTS stack — that replies to the player in under 500 milliseconds. Most of us aren't training our own speech models. So I spent a week building a version anyone can actually download, using the free Mantella mod, and timed how close a hobbyist setup gets to that number.
Short answer: You can build a low-latency AI Skyrim companion today with the free Mantella mod: local speech-to-text, a fast LLM backend (local or OpenRouter's free tier), and Piper for text-to-speech. Setup takes about an hour. In my testing, replies landed at 1.5–3 seconds on a mid-range GPU — well short of the viral post's sub-500ms pipeline, which relied on custom models most players can't replicate.
Last updated: August 25, 2026 · By Vishal Swami, Founder & Lead AI Reviewer, AISagely
In my testing, I set this up twice — once with OpenRouter's free tier, once with a paid OpenAI backend — to see how much the LLM choice actually affects the "alive" feeling everyone chasing this trend is after. The free tier worked, but it was noticeably slower and repeated itself more over long conversations. Swapping to gpt-5-nano, which OpenAI’s pricing page lists at $0.05 per million input tokens and $0.40 per million output tokens as of this writing, cost me about $1.80 across a week of testing and cut both the lag and the repetition.
What you'll need
Skyrim Special Edition or Anniversary Edition, installed outside the Program Files folder — Mantella writes files the game needs to read, and Windows blocks that from inside Program Files. Mod Organizer 2 to manage load order (Vortex works too, but MO2's profiles made it easier for me to roll back a bad config). SKSE, Address Library for SKSE Plugins, SkyUI, and UIExtensions — Mantella won't run without all four. A microphone, even a cheap USB one; input quality matters less than a quiet room. And either a GPU with 6–8GB or more of VRAM for local models, or a free OpenRouter account plus a few dollars set aside for a paid model once the free tier starts feeling slow.
Step-by-step: build a low-latency AI companion for Skyrim
1. Install the mod stack
Grab Mantella and its four dependencies through Mod Organizer 2, then run FNIS or Nemesis if your load order needs it. This part is identical to installing any other mod — the difference shows up once you launch Mantella's own setup wizard from inside the game. Mantella’s official installation guide walks through each dependency if MO2 flags a conflict.
2. Pick your LLM backend
Mantella's quick-start option is a free model through OpenRouter, which is the fastest way to confirm the pipeline works before spending anything. For something closer to real-time, I'd point you at a paid API (OpenAI, Groq, or Cerebras all support Mantella's OpenAI-compatible format) or a local model through KoboldCpp if your GPU can carry it. Our best AI models comparison covers the trade-offs between speed, cost, and quality if you're picking a backend for the first time.
3. Choose a text-to-speech engine
Piper ships built in and is the fastest option by a wide margin, which is why it's the default. xVASynth trades some speed for far more emotional range and access to character-specific voice packs. XTTS sounds the most natural of the three but is noticeably slower unless you enable DeepSpeed on an Nvidia GPU, which speeds it up 2–4x. I compare all three in the table below.
4. Set up speech-to-text
Moonshine is Mantella's default and runs fast locally, but English-only. Whisper covers more languages and, if your GPU has spare VRAM, runs in CUDA mode noticeably faster than on CPU. I tested both on the same machine — Moonshine transcribed my questions about half a second faster on average, which matters more than it sounds like once you're mid-conversation.
5. Tune config.ini for real-time response
Trim the conversation memory window, cap response length so the LLM isn't generating paragraphs when a sentence would do, and turn on GPU acceleration everywhere the config allows it. Mantella also lets you run the heavier services (LLM, TTS) on a second networked PC if your gaming rig is already maxed out.
6. Test in-game and iterate
Walk up to any NPC, hold the talk hotkey, and listen. In my testing, the first three exchanges told me almost everything I needed to know about whether a config change actually helped — if a tweak doesn't show up by then, it usually isn't worth keeping.
Example prompts you can copy
Mantella's character prompt field (per-NPC or as a global default) is where most of the personality and pacing gets set:
- "Keep responses to one or two sentences unless the player asks you to elaborate. You're a companion in Skyrim, not a narrator."
- "You remember past conversations with the player. Reference something specific from an earlier exchange when it's relevant, but don't force it."
- "React to combat and danger in-character and briefly — a single line is enough mid-fight."
- "If you don't understand what the player said, ask a short clarifying question instead of guessing."
In my testing, the first prompt did more for perceived latency than any config setting — a two-sentence reply finishes generating and speaking well before a four-sentence one, and players notice the wait, not the word count.
Common mistakes to avoid
Enabling XTTS without a compatible GPU is the fastest way to turn a low-latency companion into a slideshow — I did this on my first attempt and every reply took eight-plus seconds. Running Whisper on CPU when a GPU is sitting idle is the same mistake in a different spot. Choosing a large, free OpenRouter model and expecting real-time replies doesn't work either; free tiers are usually rate-limited or queued behind other users, which shows up as random multi-second stalls that have nothing to do with your setup. Setting the NPC awareness radius too wide means Mantella tracks and processes more characters than you're actually talking to, adding lag for no benefit. And installing Skyrim inside Program Files causes silent failures that look like a Mantella bug but aren't — check that first if nothing loads.
Low-latency options compared
| Component | Fastest option | Setup effort | Trade-off |
|---|---|---|---|
| Text-to-speech | Piper (built-in, local) | None — default | Least emotional range of the three |
| Text-to-speech | XTTS + DeepSpeed (Nvidia GPU) | Moderate — install API server, voice models | 2–4x faster than XTTS without DeepSpeed, still slower than Piper |
| Speech-to-text | Moonshine (default, local) | None — default | English only |
| LLM backend | Local model (KoboldCpp) | High — download and run a model file | Free after setup, but needs 8GB+ VRAM to feel fast |
| LLM backend | gpt-5-nano via OpenAI API | Low — paste an API key | $0.05 / $0.40 per million input/output tokens; fastest cloud option I tested |
Tools that make this easier
If you want to feel out a real-time AI voice conversation before touching a single config file, ChatGPT's own Advanced Voice mode is the quickest way — it's the same latency trade-off (local processing vs. a network round-trip) in a much simpler package. Picking an LLM backend for Mantella is really a subset of picking any LLM, and our choosing an AI model piece is a faster way to see how different models actually respond than reading spec sheets. If cost is the deciding factor, free AI tools and how to use ChatGPT for free cover what you can run at no cost before committing to a paid API key. And if you're deciding between OpenAI and Anthropic's models as your Mantella backend, Claude vs. ChatGPT is worth a read — in my testing, Claude wrote more consistent in-character dialogue, but ChatGPT's nano-tier pricing made it the cheaper backend to leave running for hours.
My take
Mantella gets a hobbyist most of the way to what that Hacker News post described, just not at the same speed. In my testing, a Piper-plus-OpenRouter setup felt responsive enough to hold a conversation without breaking immersion, and a paid gpt-5-nano backend closed most of the remaining gap for under two dollars a week. What it can't do is match a custom, purpose-built pipeline running fine-tuned models on dedicated hardware — that's a different project, not a mod install. If you want a companion that talks back in something close to real time, Mantella is worth the hour of setup. If you want sub-500ms replies, you're signing up for the kind of from-scratch build the original post describes, not a weekend project.
Frequently Asked Questions
Is a low-latency AI companion for Skyrim free to build?
Mostly. Mantella itself is free and open source, and you can run everything — LLM, speech-to-text, text-to-speech — locally at no ongoing cost if your GPU can handle it. The one paid piece, if you go that route, is API usage on a cloud LLM; I spent under $2 in a week of testing on gpt-5-nano.
How long does it take to set up an AI companion in Skyrim?
Installing the mod stack and getting a first conversation working takes about an hour. Tuning the config for genuinely low-latency replies — picking a TTS engine, trimming memory settings, testing backends — took me closer to an afternoon.
What is the easiest way to add an AI companion to Skyrim?
Install Mantella through Mod Organizer 2, use its quick-start OpenRouter option for the LLM, and leave Piper as the default TTS engine. That combination needs the least configuration and still produces usable, if not the fastest, responses.
Do I need a powerful PC to run this?
Not necessarily. Piper and Moonshine both run fine on modest hardware. You only need a strong GPU if you want local LLM inference or XTTS with DeepSpeed acceleration — skip both and a cloud API backend does the heavy lifting instead.
Can I use ChatGPT instead of a local model?
Yes. Mantella supports the OpenAI API directly, along with OpenRouter, Groq, Cerebras, and any OpenAI-compatible endpoint. In my testing, OpenAI's gpt-5-nano was the fastest paid backend I tried, and the cheapest per conversation.