Skip to Content
Loremind Unity SDKVoice Input

Voice Input

Let players speak to NPCs instead of typing. Voice input is optional - LoreMindNPC works without it.

Prerequisites

  • A working LoreMindNPC in your scene — see Your First NPC
  • For the built-in engine: the whisper.unity package (com.whisper.unity)

Overview

The SDK includes built-in support for Whisper via whisper.unity, a local speech-to-text engine. You can also bring your own STT solution by implementing the ISpeechToTextProvider interface.

OptionTypeNotes
WhisperLocalFree, offline, multi-language
CustomAnyImplement ISpeechToTextProvider

Using Whisper (Built-in)

1. Install whisper.unity

Download and import the whisper.unity package. The SDK detects it automatically and enables its Whisper integration (the LOREMIND_WHISPER define is managed for you - no need to touch Player Settings).

2. Enable in Project Settings

  1. Open Window > LoreMind > Control Panel
  2. Go to Voice tab
  3. Enable Speech-to-Text
  4. Set STT Provider to Whisper
  5. Set Language (e.g., “en” for English)

Quick Setup

Add the Component

  1. Create a GameObject (or use your Player)
  2. Add Component > LoreMind > Voice > Voice Input

The component auto-adds required dependencies:

  • MicrophoneInputHandler - Captures audio
  • SpeechCaptureController - Manages recording and transcription

Wire to NPC

Option A: Inspector

Set Target NPC to your LoreMindNPC component. Transcriptions automatically trigger NPC responses.

Option B: Code

voiceInput.OnTranscription.AddListener(text =>
{
    npc.Respond(text);
});

Capture Modes

Push-to-Talk (Default)

Player holds a key to record, releases to transcribe:

voiceInput.PushToTalk = true;
voiceInput.PushToTalkKey = KeyCode.V;

Input System note: KeyCode-based push-to-talk relies on the legacy Input Manager. If your project’s Active Input Handling is set to Input System Package (New) only, drive capture from your own InputAction handlers by calling StartCapture() / StopCapture() directly (max-duration enforcement still applies).

Manual Mode

Control capture from code:

voiceInput.PushToTalk = false;
 
// Start/stop manually
voiceInput.StartCapture();
voiceInput.StopCapture();
 
// Or capture for a specific duration (adjust based on expected speech length)
string text = await voiceInput.CaptureAndTranscribeAsync(maxDuration);

Complete Example

using UnityEngine;
using UnityEngine.UI;
using Peek.LoreMind;
using Peek.LoreMind.Voice;
 
public class VoiceNPCDemo : MonoBehaviour
{
    [SerializeField] private LoreMindNPC npc;
    [SerializeField] private LoreMindVoiceInput voiceInput;
 
    [Header("UI")]
    [SerializeField] private GameObject recordingIndicator;
    [SerializeField] private Text subtitleText;
 
    void Start()
    {
        // Configure
        voiceInput.PushToTalk = true;
        voiceInput.PushToTalkKey = KeyCode.V;
 
        // Wire up events
        voiceInput.OnCaptureStarted.AddListener(() =>
        {
            recordingIndicator.SetActive(true);
            subtitleText.text = "Listening...";
        });
 
        voiceInput.OnCaptureStopped.AddListener(() =>
        {
            recordingIndicator.SetActive(false);
            subtitleText.text = "Processing...";
        });
 
        voiceInput.OnTranscription.AddListener(text =>
        {
            subtitleText.text = $"You: {text}";
            npc.Respond(text);
        });
 
        voiceInput.OnError.AddListener(error =>
        {
            subtitleText.text = $"Error: {error}";
            recordingIndicator.SetActive(false);
        });
 
        // Show NPC response
        npc.OnResponseReceived.AddListener(response =>
        {
            subtitleText.text = $"NPC: {response}";
        });
    }
}

Configuration

Capture Settings

SettingDescriptionDefault
Max Recording SecondsAuto-stop after this duration10s
Min Recording SecondsDiscard shorter recordings0.3s

Auto-Response Settings

SettingDescriptionDefault
Target NPCNPC to send transcriptions toNone
Minimum Text LengthDiscard shorter transcriptions3 chars

Events

// Capture lifecycle
voiceInput.OnCaptureStarted.AddListener(() => ShowRecordingUI());
voiceInput.OnCaptureStopped.AddListener(() => HideRecordingUI());
 
// Results
voiceInput.OnTranscription.AddListener(text => HandlePlayerSpeech(text));
voiceInput.OnError.AddListener(error => ShowErrorMessage(error));

API Summary

Properties

bool IsCapturing { get; }
float CaptureDuration { get; }
bool IsTranscribing { get; }
string LastTranscription { get; }
 
bool PushToTalk { get; set; }
KeyCode PushToTalkKey { get; set; }
LoreMindNPC TargetNPC { get; set; }

Methods

void StartCapture()
void StopCapture()
Task<string> CaptureAndTranscribeAsync(float durationSeconds = 0f)
void CancelCapture()

Audio Providers

UnityMicrophone (Default)

Uses Unity’s built-in Microphone class. Works everywhere Unity supports microphone input.

Dissonance (Multiplayer)

For multiplayer games using Dissonance Voice Chat: add the DissonanceAudioProvider component — it captures audio from Dissonance instead of the microphone.

Custom Provider

Implement IAudioProvider for custom audio sources:

public class CustomAudioProvider : MonoBehaviour, IAudioProvider
{
    public event AudioCaptureEventHandler OnAudioCaptured;
    public bool IsCapturing { get; private set; }
 
    public void StartCapture(int sampleRate)
    {
        // Start capturing from your audio source
    }
 
    public void StopCapture()
    {
        // Stop and fire OnAudioCaptured with captured data
    }
}

Platform Support

Voice input uses microphone capture and is not available on WebGL. All other Unity platforms with microphone access are supported.

Best Practices

Provide Visual Feedback

Always show recording and processing states:

voiceInput.OnCaptureStarted.AddListener(() => recordingIcon.SetActive(true));
voiceInput.OnCaptureStopped.AddListener(() =>
{
    recordingIcon.SetActive(false);
    processingIcon.SetActive(true);
});
voiceInput.OnTranscription.AddListener(text => processingIcon.SetActive(false));

Display Transcriptions

Show players what was heard:

voiceInput.OnTranscription.AddListener(text =>
{
    subtitleText.text = $"You: {text}";
});

Offer Both Voice and Text

// Voice input
voiceInput.OnTranscription.AddListener(SendToNPC);
 
// Text input
textInput.onEndEdit.AddListener(SendToNPC);
 
void SendToNPC(string playerInput)
{
    if (!string.IsNullOrEmpty(playerInput))
        npc.Respond(playerInput);
}

Custom STT Provider

If you prefer a different speech-to-text solution, implement the ISpeechToTextProvider interface:

using Peek.LoreMind.Services;
using System.Threading.Tasks;
 
public class MyCustomSTT : ISpeechToTextProvider
{
    public string ProviderName => "MyCustomSTT";
    public bool IsReady { get; private set; }
 
    public async Task<bool> InitializeAsync()
    {
        // Initialize your STT engine
        IsReady = true;
        return true;
    }
 
    public async Task<string> TranscribeAsync(
        float[] audioData,
        int sampleRate,
        int channels)
    {
        // Transcribe audio using your STT engine
        // audioData: raw float samples
        // sampleRate: typically 16000 for STT
        // channels: 1 for mono, 2 for stereo
 
        // Return transcribed text
        return "transcribed text";
    }
 
    public void Dispose()
    {
        // Clean up resources
    }
}

Troubleshooting

No microphone detected

Check microphone permissions:

  • Windows: Settings > Privacy > Microphone
  • macOS: System Preferences > Security & Privacy > Microphone
  • Mobile: App must request microphone permission

Transcription returns empty

  • Verify recording wasn’t too short (below MinRecordingSeconds)
  • Check Whisper model loaded correctly
  • Speak louder or closer to microphone

”Speech-to-text not enabled”

Open Window > LoreMind > Control Panel > Voice and enable Speech-to-Text.

First transcription is slow

Whisper runs locally. First transcription loads the model. Subsequent transcriptions are faster.

Next Steps

Last updated on
Context SystemLore Sync