AI Speech Recognition in Unity

Introduction

This tutorial guides you through the process of implementing state-of-the-art Speech Recognition in your Unity game using the Hugging Face Unity API. This feature can be used for giving commands, speaking to an NPC, improving accessibility, or any other functionality where converting spoken words to text may be useful.

To try Speech Recognition in Unity for yourself, check out the live demo in itch.io.

Prerequisites

This tutorial assumes basic knowledge of Unity. It also requires you to have installed the Hugging Face Unity API. For instructions on setting up the API, check out our earlier blog post.

Steps

1. Set up the Scene

In this tutorial, we'll set up a very simple scene where the player can start and stop a recording, and the result will be converted to text.

Begin by creating a Unity project, then creating a Canvas with four UI elements:

1. Start Button: This will start the recording.
2. Stop Button: This will stop the recording.
3. Text (TextMeshPro): This is where the result of the speech recognition will be displayed.

2. Set up the Script

Create a script called SpeechRecognitionTest and attach it to an empty GameObject.

In the script, define references to your UI components:

[SerializeField] private Button startButton;
[SerializeField] private Button stopButton;
[SerializeField] private TextMeshProUGUI text;

Assign them in the inspector.

Then, use the Start() method to set up listeners for the start and stop buttons:

private void Start() {
    startButton.onClick.AddListener(StartRecording);
    stopButton.onClick.AddListener(StopRecording);
}

At this point, your script should look something like this:

3. Record Microphone Input

Now let's record Microphone input and encode it in WAV format. Start by defining the member variables:

private AudioClip clip;
private byte[] bytes;
private bool recording;

Then, in StartRecording(), using the Microphone.Start() method to start recording:

This will record up to 10 seconds of audio at 44100 Hz.