Beginner's guide

Start here if you have not installed a Unity package with external dependencies before. The first goal is one typed question and one text answer in Hello NPC. After that, add voice, microphone input or your own character, then make a player build. Download and setup time depend on your connection, hardware and the features you choose.

After installation: follow the first text dialogue steps in one setup window.

Choose a speech setup for your game

Start with typed dialogue. Add Whisper when players need a microphone, then choose a ready voice:

The built-in Qwen speech presets use the 0.6B CustomVoice Q4 talker and Q8 codec. New Qwen engine components use the bundled CPU DLL with External server URL and Style empty, and the existing 1.5-second lead buffer. Serena (serena) is the default for every built-in Qwen language and preset. You can choose another of the nine ready speakers; creating a voice from a description (VoiceDesign) is not part of this version. Delivery instructions require a separately checked 1.7B CustomVoice setup. Check the language, voice terms and response time on your players' hardware; see Languages, Hardware tiers and Licensing.

Model Manager downloads the speech model files. The package includes a Windows x64 CPU Qwen native DLL; leave External server URL empty to use it. The ordinary Qwen route needs no manually started HTTP server, executable or compiler setup. Keep an existing working Piper route. See the native speech setup and Licensing.

0. What this is, in one paragraph

With the default local configuration, your character talks. The language model, the speech recognition and the speech synthesis all run on the player's own machine — no cloud, no API keys, no per-token bill, no telemetry, and it works with the network cable pulled out. For text dialogue you install a model runner, one model file and this plugin. Voice and microphone input add their own optional engines and files. One editor-only exception is yours to choose: the optional MCP server hands project data to the coding agent you connect, and a cloud-based agent sends it on to its provider — see Agent tools.

First milestone: compatible LLM for Unity → Set up first dialogue → one model → Play → a typed question and a text answer. Voice and microphone input are optional next steps.

1. Before you start

If any of those are a problem, read Is this for your game? before installing the dependencies or downloading large files.

2. Make a project

New project, any template. Nothing here depends on which.

You can also use an existing project. Importing the package adds its assets; opening the sample may ask you to save the current scene. Setup changes and build inclusion are explicit actions. Keep your existing scene and model/voice bindings when adapting the sample.

3. Install LLM for Unity — required

This is the thing that actually runs the language model. Without it your character cannot form a sentence.

• Package id: ai.undream.llm, version 3.0.3 or newer
• Free, and not bundled here — it is somebody else's package under its own licence, and freezing a copy inside this one would pin you to whatever version shipped that day
• Install it the way its own documentation says

How you know it worked: Window ▸ Package Manager lists LLM for Unity.

If Unity was already open, it may take one recompile before this plugin notices. You do not need to restart — Tools ▸ Offline AI NPC ▸ Advanced ▸ Re-check LLMUnity installation forces the check.

4. Import Offline AI NPC

Import the package. In Welcome, choose Set up first dialogue. You can reopen Welcome with Tools ▸ Offline AI NPC ▸ Welcome, or go directly to Tools ▸ Offline AI NPC ▸ Set up first dialogue.

5. Set up the first text dialogue

1. In First dialogue, if Hello NPC in the scene is empty, click Open Hello NPC sample. Otherwise keep or select the character you want to configure.
2. Click Choose this model. If its file is missing, Model Manager opens for that model. Download it there, then click Return to first dialogue.
3. Click Preview changes, read the proposed assignment, then Apply to this Hello NPC. You can restore the previous assignment with Unity Undo.
4. Save the scene, press Play and wait for startup. Type a question, press Enter and read the answer in the subtitle.

Already have a GGUF? Expand Use an existing local file and select it. To import a file, click Open Model Manager; see the own-GGUF steps.

Choose another catalog option lets you filter the model catalog by language and configuration size. The character's dialogue language is set in its persona settings; see Languages.

Start with text. Optional speech setup and the later voice/microphone sections can wait until this works. For distribution, check Licensing; advanced tier startup is in Hardware tiers.

6. Check the text setup

If the model does not start, open Tools ▸ Offline AI NPC ▸ Check my setup with Hello NPC open. Resolve the LLM dependency, selected model and scene-reference items for text dialogue.

Voice, codec and speech-output items belong to optional audio. See Getting started for more detailed setup help.

7. Your first typed conversation

If it is not already open, click Open Hello NPC sample in Welcome and save the current scene if prompted. It opens the already-wired Samples/HelloNpc/HelloNpc.unity. Press Play and wait for startup to finish. Type a question into the text box, press Enter and read the answer in the subtitle. If the model is unavailable or another turn is busy, the question stays in the input field; submit it again after resolving the message. It is not queued automatically. No microphone or ready speech engine is required for this first typed turn. A voice warning points to optional audio setup.

A healthy default tier start prints these lines. Learn to recognise them:

[OfflineAINPC] Hardware: NVIDIA GeForce RTX 3060 Laptop GPU (5996 MB VRAM, discrete), ...
[OfflineAINPC] Tier resolved: Standard — 3996 MB VRAM free after a 2000 MB game reserve
[OfflineAINPC] Running qwen3-4b-q4km at tier Standard with 36 GPU layers
[OfflineAINPC] Startup verdict: tier Standard — ...; model qwen3-4b-q4km, 36 GPU layers

For the tier route, the startup verdict summarizes the selected tier and model. It does not prove voice readiness or replace asking the first typed question. If text never arrives, inspect the Console and LLM/model setup rows, then Troubleshooting. The own-GGUF route disables its linked TierBootstrap; use the assigned LLM's startup diagnostics rather than expecting a tier verdict.

For Qwen3-TTS's ordinary DLL route, look for Native CPU ready in the Console; Piper has its own engine diagnostics. An explicitly configured external server has separate readiness diagnostics. Neither speech engine is required for text dialogue.

Startup and the first reply can take longer than later turns, including while a prompt cache is being filled. This is a possible cause of delay, not a guarantee that an unresponsive scene is healthy. Check the startup verdict for tier startup, or the assigned LLM startup for the own-GGUF route, plus the Console and setup report.

Normal, or broken?

These behaviours can be expected in the stated conditions. Use the setup report and Console to distinguish them from a failed or still-running startup:

And these are not — each has a page that names the cause:

8. Optional: add a voice

The package includes the Windows x64 CPU Qwen speech library. Its target CPU must support AVX2, BMI2, FMA, F16C and SSE4.2. macOS and Linux libraries are not included.

1. In Tools → Offline AI NPC → Model Manager, download the 0.6B CustomVoice Q4 talker and Q8 codec under Voices and speech models (about 0.9 GB together).
2. Use the scene's Qwen3-TTS Engine through TTS Engine Manager and its language binding. The sample supplies this wiring; a custom character must select that engine.
3. Leave External server URL and Style empty. Use serena in Default voice to keep the same voice across languages. A character's SpeechRequest or Speaker voices entry can override it. Set the language separately; choosing a voice does not translate the text.
4. In Play, type a short prompt and check the reply's subtitle and speech in the chosen language. Repeat this check in your built Windows player with the model files available.

The default route loads the DLL in Unity and streams PCM to the existing speech session. It does not start tts-server.exe. The native profile is CPU-only with eight threads. StopAsync waits for native generation and worker settlement before releasing the context. Cancellation requests cooperative cancellation; it is not an instant-stop guarantee.

A separately launched server and External server URL / AttachTo(url) are optional advanced alternatives. Keep that server's own model, backend, version and distribution requirements separate.

For a new Piper setup, put piper.exe for Windows and its companion files in Assets/StreamingAssets/piper/, and the name.onnx / name.onnx.json pair in Assets/StreamingAssets/piper/voices/. Other platforms remain unverified. Preserve existing working configurations and custom paths.

Piper remains supported alongside Qwen3-TTS. Keep both files, name.onnx and name.onnx.json; an incomplete pair is refused with a diagnostic. Model Manager's Import file copies a local pair while preserving the originals and existing destination files. The imported pair goes into Model Manager's ModelsRoot; this does not install the engine or connect the built-in Piper driver to that voice. Copying and verification show inline background progress; Cancel import requests safe cleanup. See Languages and Troubleshooting.

Run Check my setup again and fix the speech rows. Ask another typed question and check both the subtitle and audio. File presence does not prove native engine startup.

9. Optional: add microphone input

This is speech input: the player speaking instead of typing. Add it after the typed path works; skip it if you do not need a microphone.

1. Install whisper.unity: Window ▸ Package Manager ▸ + ▸ Install package from git URL, and paste:

`` https://github.com/Macoron/whisper.unity.git?path=/Packages/com.whisper.unity ``

1. In the Model Manager, open Speech recognition and download the Whisper model your preset names — ggml-tiny.en.bin or ggml-base.en.bin for English on Light and Standard, the multilingual ggml-base.bin or ggml-small.bin otherwise. An .en model recognizes English only.
2. Select the Hello NPC character and add two components: **Add Component ▸ Offline AI NPC ▸ Push To Talk and Add Component ▸ Offline AI NPC ▸ Whisper Speech Recognition**. For runtime English/Russian switching, download a multilingual model such as ggml-small.bin and set Model File Name to it before entering Play. The loaded model stays in use for the session; a Language Definition's STT model id does not replace it.
3. Press Play, click outside the text box, hold V, speak and release. What it heard goes to the character as if typed; the screen says when a press was not sent, and why.

A missing dependency is never a compile error here. Without whisper.unity the Whisper Speech Recognition component is compiled out; Push To Talk stays, and with no recognizer beside it every press is reported as not sent. Your own recognizer can implement ISttEngine instead. Details: Getting started.

10. What is in the plugin, and what each part is for

Everything below ships in the package. You will not need all of it on day one — most projects start with dialogue and add the rest when they meet the problem it solves. This section exists so that you know what is already there before you build it yourself.

The runtime, by area

Parts worth knowing about early

The utterance gate (UtteranceGate). Whisper hallucinates confidently on silence -- "Thank you for watching", "Subtitles by …", a lone full stop — because it was trained on video with end cards. A push-to-talk key brushed by accident therefore looks exactly like a player speaking. The danger is not a stray subtitle: if your game listens for a word as a command, an invented one triggers it, and a key brushed in passing sends the player somewhere they never asked to go. The gate judges the audio first and the text second, and every rejection says which one it was.

Action repair (ActionRepair). When the model returns a malformed action, one retry is allowed — and it asks for the action alone, never the reply again. By the time anyone knows the action was bad the speech has already been spoken, and regenerating it would make the character say the same thing twice.

Traits versus state (TraitPreset versus StateAxes). Traits never move: they are why two characters with identical axes behave differently — the shy one starts lower and warms slowly. State moves. Mood is deliberately not persisted, because a feeling that outlives the session is not a feeling, it is a trait.

Perception weights (PerceptionWeights). Serialized data, never constants. The right balance of attention for a cluttered room is not the right balance for an open field, and that is a designer's call rather than the plugin's.

The language directive (LanguageDirective). The plugin's own contribution to your system prompt: answer in THIS language, and keep it about this long. Both sentences are written in the language they are asking for — an instruction the model cannot read is not an instruction. Your dialogue code appends it: LanguageDirective.For(LanguageRuntime.Current) returns what the language definition authors, and nothing when it authors neither sentence. Hello NPC speaks the language of its own system prompt and does not add it.

Preset choice (PresetCatalog). Your entry outranks an official one for the same slot; otherwise the exact tier wins, then the closest lighter tier — because running something smaller beats running nothing.

The editor windows

All under Tools ▸ Offline AI NPC.

Every item in the menu

Tools ▸ Offline AI NPC, top to bottom — the windows first, everything else under Advanced:

What the plugin deliberately does not do

Knowing these saves you looking for them:

• Qwen speech includes its Windows x64 CPU native DLL. Model files are downloaded separately. Piper is still an external program you install; preserve its working executable, voice/config pair and binding.

11. Your own character

Start from the canonical Samples/HelloNpc/HelloNpcCharacter.cs after the sample answers. It contains the real LLMUnity call, the check that refuses a turn while another runs, action handling and cleanup. Preserve its public Say(string) entry point and serialized references when adapting an existing scene.

The following is a teaching outline, not a copyable controller or a model backend:

typed question -> one admitted turn -> streamed LLM reply
              -> local parser -> local speech session -> subtitle (+ optional audio)
completion    -> complete parser -> InputComplete -> dispose session
disable       -> invalidate callbacks -> cancel generation -> dispose session

For guarded wiring in C#, read Dialogue. Its GenerateReply, CancelGeneration and ShowSubtitle methods are explicit placeholders for your existing backend and UI; the empty GenerateReply does not call a language model. Use Hello NPC for a runnable starting point rather than filling these seams on day one.

The four checks to preserve

1. A turn owns its callbacks and _busy. Keep parser and speech references local to that turn and reject callbacks after disable or replacement. Release the lock only while that turn is current. OnDisable must cancel the backend and dispose speech explicitly: Unity can stop coroutines without running finally.
2. A null or unready engine is normal for text dialogue. Fetch the current engine through a Func<>; speech is optional and the session can deliver subtitles without audio. Diagnose a failed voice setup before testing speech.
3. Cumulative versus delta. LLMUnity's callback carries the whole reply so far; use AppendCumulative. A backend streaming true deltas needs Append.
4. Completion and cleanup. Complete the parser and call InputComplete() when generation ends. Dispose the session on failure or disable, and prevent an old continuation from clearing a newer turn's state.

Memory storage is explained in Dialogue; moods and relationships are in State, which opens with a ten-line version.

12. The names you will actually type

The section above says what each area is for. This one says what to reach for, because a concept you cannot name is a concept you cannot look up. Only the types you touch are listed; the rest are internals that happen to be public.

Put on a GameObject

Implement in your own code

Call

Author as assets

Configure in the inspector or in code

Coverage, honestly. The documentation names the types you put on objects, implement, call and author; the rest are data holders, enums and internals that happen to be public. If you meet a name that no page explains, that is a gap in these docs rather than something you are expected to already know.

13. Making a build

The step most likely to surprise you.

Models are excluded by default — every one of them. That default is a feature: six models downloaded while experimenting would otherwise silently become a twenty-gigabyte build, discovered at upload time. Turn on the one you ship, in the Model Manager.

An included model is staged into Assets/StreamingAssets/OfflineAINPC for the duration of the build and removed afterwards. This is the plugin's staging path for selected model files; it avoids leaving those copies in StreamingAssets between builds. For the own-GGUF route, configure the player model path separately: including the file does not rewrite LLM._model. See the path requirements.

When Block a build whose scenes reference a model file that is not on disk is enabled in Model Manager (BlockBuildOnBrokenReferences), detected missing model references stop the build; disabling it reports a warning. The validator matches file names in its search roots. It does not verify player paths, model compatibility or portability.

• Qwen speech includes its Windows x64 CPU native DLL. Model files are downloaded separately. Piper is still an external program you install; preserve its working executable, voice/config pair and binding.

Before you send it to anyone

• run it on a machine that is not your dev machine, ideally without Unity installed;
• find the startup verdict for tier startup, or check the assigned LLM startup and model path for the own-GGUF route, in the player log;
• type one question and read one answer; if you enabled audio or microphone input, test those separately.

14. When it does not work

In the order these actually happen:

Troubleshooting maps real error strings to fixes.

15. Words you will meet