The goal hasn't changed since day one: a voice assistant for my room that I build and control myself, instead of an Alexa or a Google Home. I say the wake word, ask something, and the house answers or acts. This page tells how it went from a DIY build in 2025 to the setup I use today, and what is still missing.
The first version was fully DIY. I took the speaker out of a cheap giveaway speaker, measured it, and designed a simple round enclosure in Fusion 360 to test the fit. Then I ordered the rest on AliExpress:
The first power-on went badly: a wiring mistake killed the ESP32. Frustrated, I left the project aside for a while, and in the meantime pulled a sock over the enclosure to hide the speaker. It actually looks pretty clean.
Once rebuilt, it did work: wake word, commands, spoken answers. But the microphone was never solid. A bare INMP441 sends the raw sound to the ESP32 with no processing at all, and sensitivity was mediocre: a few meters away or with music playing, recognition became unreliable. No amount of software on the ESP32 fixes a bad audio signal, so I had to change the hardware.
For the second version I stopped building the audio front-end myself and took a board designed for it: the Seeed Studio ReSpeaker Lite, sold with a XIAO ESP32-S3 already soldered on its back.
What makes the difference is the XMOS XU316 chip sitting between the two microphones and the ESP32. It processes the sound on the board before anything is sent: interference cancellation, acoustic echo cancellation (it hears me even while it plays music), noise suppression and automatic gain control. The ESP32 receives a clean signal over I2S, on two channels: one tuned for speech recognition, the other for wake word detection. Seeed rates it for voice pickup up to about 3 meters, and compared to the INMP441 it's night and day.
The board also carries the audio codec (TLV320AIC3204), the speaker amplifier, a 3.5 mm jack, an RGB LED and two buttons, all powered by a single USB-C cable. Parts for this version:
I modeled a new case around the ReSpeaker Lite, in two parts: the body and a lid.
The front panel is tilted toward the room so the microphones face me instead of the ceiling. It carries the brain logo, openings for the two buttons and pinholes for the microphones. The speaker fires through the grille in the lid, and the ports come out on the side.
The printed version, opened: the board stands against the front panel, the speaker sits at the bottom and the LED strip runs along the top edge.
The strip is both an indicator and a bit of fun: at a glance I know where the request is.
The brain logo lights up too: the wall behind it is thinner, so the board's RGB LED shines through. A full request, from "Okay Nabu" to the answer:
"Okay Nabu, ..."
|
ReSpeaker Lite + XIAO ESP32-S3 (ESPHome)
|
| Wi-Fi
|
Home Assistant (Raspberry Pi 5)
|
|-- 1. speech-to-text : Scaleway, whisper-large-v3
|-- 2. understanding : local intents, otherwise Gemini
+-- 3. text-to-speech : Piper, local (fr_FR-siwis-medium)
|
answer played on the speaker
The wake word is detected on the ESP32-S3 itself by microWakeWord: nothing is streamed before it triggers, and saying it again cuts the answer.
For speech-to-text, Home Assistant uses the OpenAI Whisper Cloud custom integration from HACS, pointed at Scaleway's OpenAI-compatible endpoint. A 3 to 6 second sentence comes back as text in about 0.3 s. A short prompt listing only my device and scene names helps Whisper spell them right; a long one made of full sentences made it worse.
Anything the local intents recognize is handled in about 0.2 s without leaving the house. The rest goes to models/gemini-3.1-flash-lite (minimal thinking) through the Google Generative AI integration. It can see the entities I expose to Assist, so it can act on the house as well as answer questions.
I picked the model by measuring, not by name. The bigger Flash models took 6 to 15 seconds or failed at peak time, and one "latest" alias answered simple questions in 0.8 s but took 16 to 28 s as soon as it had to call a Home Assistant tool. Flash-Lite answers a general question in 1 to 2 s, and 2 to 6 s when it acts on a device, because that takes two round trips: the model asks for the tool, Home Assistant runs it, the model phrases the answer. Every exposed entity is also sent with each request, so I keep that list short.
The system prompt matters as much as the model. The answer is read aloud, so: one short sentence, no lists, no markdown, no emoji. Spoken device names are always approximate, so it must pick the closest device and act instead of saying it doesn't exist. And it must answer general questions from its own knowledge instead of replying "search the internet".
Scaleway wasn't my plan. I wanted the transcription at home, so I first ran Whisper locally, on the Raspberry Pi 5 and then on the Intel N100 mini PC of my rack: 8.75 s to transcribe 3.5 s of speech. Nobody waits nine seconds for a light to turn on.
Getting large-v3 (about 1.5 billion parameters) to answer in a fraction of a second takes a real GPU. Keeping one powered 24/7 for a model that works a few seconds a day doesn't make sense: even with a hundred requests a day, it would be busy less than 1% of the time.
Scaleway hosts the model in Paris, billed on the audio length (€0.003 per minute), with a free tier of 60 minutes of transcription that I'm using for now. The plan is to simply pay for it and keep it there: a French provider, data processed in Europe, and a few euros a month at most.
The satellite runs ESPHome on ESP-IDF. The ReSpeaker-specific parts come from formatBCE's ESPHome integration (the one Seeed's wiki points to): a fork of the i2s_audio component and a respeaker_lite component that also flashes the XMOS firmware (v1.1.0) from the ESP32.
Audio runs at 48 kHz / 32 bits with the XMOS as I2S master. The output goes through a mixer with two pipelines, one for announcements and one for media, so the music is lowered by 20 dB while the assistant talks.
Each LED strip state is an ESPHome script called from the voice_assistant events, so the whole look is defined in one place.
Before the strip, I tried a 1.28" round GC9A01 screen playing a pixel-art galaxy animation on the wake word, which is where the device name "ReSpeaker Galaxy" comes from. It worked, but the ReSpeaker board uses almost every pin of the XIAO: only GPIO3 and GPIO4 are free, so the screen had to take over the RGB LED's pin. I put it aside for now.
The best part is that I can ask it pretty much anything and it works. A few examples (it speaks French):
RMT TX timeout / invalid argument, never mentioning memory. Fix: 48 for the onboard LED, 144 for the strip.on_timer_* handler exists in the YAML.503 UNAVAILABLE ("This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later."), and the free tier's per-minute quota answers 429 RESOURCE_EXHAUSTED when requests come in bursts. Home Assistant then reads the raw error out loud, in English. It needs a retry, a fallback model and a short French message.