nemotron-halloween

Building a Haunted House with NVIDIA Nemotron Open Models

Or rather…

Boo! Now Get Off My Lawn!

TL;DR: MonsterHouse is an interactive projection I created for Halloween. He listens to trick-or-treaters, answers in the voice of a grumpy old pirate ship and softens when visitors remember their manners. He is built on the NVIDIA Nemotron Voice Agent with NVIDIA Nemotron ASR, Nemotron 3.5 Bolt E4B 4GB, Magpie TTS Zeroshot and Audio2Face-3D, and he runs on a single workstation.

I wanted the kids visiting my house at Halloween to interact in a fun and unique way. The house was going to be in charge and giving out the treats, but more grumpy than scary. Somehow, my solution was to give the garage door a mouth and teach him to complain.

That became MonsterHouse: a grumpy old building with pirate stories, a fiery grin and a suspicious weakness for “please.” NVIDIA Nemotron Voice Agent supplies the conversation foundation. Nemotron Speech does the listening, Bolt E4B is the brain, NVIDIA Magpie TTS Zeroshot is the live voice, and NVIDIA Audio2Face-3D makes the wooden mouth talk.

How Monster House works

• Listening: NVIDIA Nemotron Speech ASR turns visitor speech into text. [CONFIRMED: nemotron-speech-streaming-en-0.6b.q8_0.gguf ]
• Thinking: NVIDIA Nemotron 3.5 Bolt E4B 4GB, an open small language model for conversational agents, writes each reply. A LoRA adapter supplies the personality.
• Speaking: NVIDIA Magpie TTS Zeroshot voices each reply from a short reference sample of the character.
• Animating: NVIDIA Audio2Face-3D turns clean speech into mouth animation in Unreal Engine 5.
• Seeing: NVIDIA Nemotron 3 Nano Omni describes costumes, while YOLO and ByteTrack count and track visitors.
• Running it: NVIDIA Nemotron Voice Agent coordinates everything on one workstation with an NVIDIA RTX PRO 6000 Blackwell Max-Q and an NVIDIA RTX PRO 4000 Blackwell.

A grin full of splinters

A few expressions from a house with plenty to say.