What happened
NVIDIA engineer Rick Grandy turned his garage door into a talking Halloween character called MonsterHouse, powered entirely by open NVIDIA Nemotron models. The system listens to trick-or-treaters, answers in the voice of a grumpy old pirate ship, and softens when visitors remember their manners. It runs on a single workstation with two NVIDIA RTX PRO GPUs.
The pipeline uses Nemotron Speech ASR to transcribe visitor speech, Nemotron 3.5 Bolt E4B 4GB, a small open language model, to generate replies, Magpie TTS Zeroshot to voice them, and Audio2Face-3D to animate a 3D mouth in Unreal Engine 5. Nemotron 3 Nano Omni, a vision-language model, describes costumes from a camera feed, while YOLO and ByteTrack count and track visitors.
Why it matters
The project shows how open models can be combined into a full multimodal agent without cloud dependence. Bolt E4B was chosen over larger models like Qwen3 30B-A3B for its smaller footprint, running quantized at 4 GB. The author trained LoRA adapters on over 5,000 multi-turn conversations to shape the character’s personality, and cached responses keep latency low: 1.70 seconds median from speech to audio.
What to watch
The MonsterHouse code itself is not packaged for release, but every model used is publicly available on Hugging Face, and the Nemotron Voice Agent blueprint provides the starting pipeline. The author notes that real-world testing with actual trick-or-treaters is still pending, and that automatic candy delivery remains a future goal.
NVIDIA’s open-model strategy is on display: a single garage can host a full voice agent, animation and vision stack.