esp32-thinks turns a round ESP32 gadget into a voice assistant with a brain you choose
A 1,600-line patch on Meta's muse-gadget-sdk that points speech-to-text, chat, and text-to-speech at any three OpenAI-compatible servers.
esp32-thinks, by raullenchai, is firmware for a small round-screened ESP32-S3 device that listens, sends your question off to a model, and reads the answer back. The interesting part is that it does not care whose model. Speech-to-text, chat, and text-to-speech are three separate endpoints, and each can point at a different provider.
The README’s summary is “a Muse that thinks with your models.”
How it works
The project is not a firmware written from scratch. It is a patch of about 1,600 lines applied to a pinned commit of Meta’s open-source muse-gadget-sdk. The SDK supplies the device framework, the UI, audio, and over-the-air updates. The patch adds what the README calls the brain session, the wake-word integration, and the tooling to build and flash it.
A turn starts one of two ways. You say “Hi, ESP”, which is detected on the chip by the esp-sr WakeNet model, or you press the knob. Wake-word turns end after about 1.4 seconds of silence. Button turns end when you let go.
After that, nothing runs on the device. The audio goes to a /audio/transcriptions endpoint, the text goes to /chat/completions with streaming, and the reply goes to /audio/speech. Any server that speaks the OpenAI API shape will do for each one. The repo ships an all-OpenAI preset and one that mixes DeepSeek for chat with Whisper large-v3-turbo for transcription.
The trick that makes it tolerable to talk to is sentence-level streaming. The firmware does not wait for the whole reply. It splits the stream at sentence boundaries, including Chinese, Japanese, and Korean punctuation, synthesizes each sentence as it arrives, and appends the audio to what is already playing. It also keeps TLS connections open between turns and trims silence before uploading.
What it costs you in time
The README is direct about latency. A warm turn takes about 5 to 7 seconds from the end of your speech to the first spoken word. The text caption shows up on the screen at about 2 to 3 seconds. Transcription alone is about 3.4 seconds on one preset, and a cold start adds about 1.5 seconds.
That is slow next to a phone assistant, and the breakdown shows why: three network round trips in a row, with transcription the largest.
Other limits from the README: only one board is supported, the Seeed SenseCAP Watcher W1-B. API keys are compiled into the firmware, which the author describes as fine for a hobby recipe and not for a product. Running against local models on your own network is on the roadmap, not in the code.
Try it yourself
The repo is MIT licensed. According to the README you need the Watcher board, a Linux or macOS machine with ESP-IDF v6.x, a clone of both this repo and muse-gadget-sdk, and API keys for whichever providers you pick. The README tells you to back up the factory NVS partition before flashing, and warns that flashing at high baud rates with plain esptool can corrupt the device; its own script flashes slowly, at 115200 baud, in about two minutes.
We have not built or flashed this ourselves; this is based on the README.