Skip to content

Personal project · shipped to external testers · Aug 2026

Nod

A desktop voice assistant and live meeting HUD

A Windows overlay that listens for a wake word, answers out loud in a locally-synthesised Philippine-English voice, and keeps a rolling summary of the meeting you're in — with microphone and speaker audio kept structurally incapable of reaching the same code path.

Python 3.10PyQt6faster-whisperPiper TTSOllamaGeminiOpenCVTesseract
intent accuracy
98.4%
p50 classification
606ms
end to end, CPU only
2–4s
lines of Python
11.4k
test suites
16
realtime local TTS
14×

01

Two halves, one overlay

Nod is a frameless, click-through, always-on-top card pinned to the top of the screen. The agent half listens to the microphone for a wake word, transcribes locally with Whisper, classifies intent and answers out loud. The meeting half captures speaker audio through WASAPI loopback, OCRs the screen, and keeps a rolling 120-second summary card of the call you're in.

It is a single Python process: five daemon worker threads plus the Qt main thread, connected only by queues. Workers never touch widgets — the HUD drains its queue on a timer on the main thread.

One nice detail: the overlay publishes its own rectangle so the screen reader can blank that region before OCR. Without it, Nod reads its own last suggestion off the screen and confirms itself in a loop.

Meeting00:12:04
listening

Rina asked whether the Q3 migration slips if the vendor contract is not signed by Friday.

Suggested · say

“Contract is with legal — I’ll have an answer by the Thursday standup.”

120s
The HUD card, rebuilt in CSS from the overlay’s own colour tokens

02

A meeting cannot give Nod orders

Microphone and speaker audio never share a queue. Two capture threads, two transcription workers, two separate Whisper models — and the queue that carries commands is fed only by the microphone.

So a webinar host saying "hey Nod, join the next call" is structurally incapable of issuing a command. Not filtered, not scored, not prompt-guarded: there is no path. That is the one rule in the codebase written down as non-negotiable.

The same instinct closed a command-injection hole at the data layer. Nod's speech is assembled from calendar titles and speech recognition, so the text-to-speech shim takes text as data on stdin and never interpolates it into a shell command. A meeting titled with a shell substitution gets read aloud instead of run.

03

The accent is a phoneme transformation

Piper's voice catalogue has no Philippine English. The options were to mangle English through a Tagalog model, or to clone a speaker. I did neither.

Instead the synthesiser is intercepted between phonemisation and audio generation, and documented sound changes are applied to the IPA directly: TH-stopping, rhoticity, no intervocalic flapping, final /z/ devoicing, no vowel reduction, TRAP lowering. No dataset, no training, no cloning — and it is testable without audio, because you can feed it IPA and assert on IPA.

The two features most associated with caricature are implemented and off by default. Every substitution target is validated against the model's own phoneme table first, because an unknown phoneme id renders as silence, and a word vanishing mid-sentence is worse than a word with the wrong accent.

04

Prompting beat fine-tuning

Local intent classification started at 87.3% zero-shot on a 3B model. Sixteen worked examples took it to 98.4% — 11.1 points for about 25 milliseconds, with zero malformed JSON across 252 calls.

The next 3.8 points came from deleting seven rows of my own dataset, not from tuning. Reading the confusion table showed the model was being punished for an ASR error I had invented rather than observed.

What I did not do was chase the last few points. The remaining errors are roughly 60% Tagalog imperatives that a 3B model will not handle, 20% genuinely ambiguous cases where no correct label exists, and 20% my own bad labels. Deleting the Tagalog rows would have pushed the headline above 97% and meant nothing.

zero-shot

87.3%

+11.1 pts · ~25ms

16 worked examples

98.4%

Local intent classification · qwen2.5:3b · 8 intents, 126 eval rowsZero malformed JSON across 252 calls, at 606ms p50.

05

The optimisation I reverted

Removing an image upscale before OCR made that stage 1.6× faster and looked obviously correct. It also retained 13% of the text — 2,556 characters against 3,548 on the same frame.

I put it back, and left the measurement in a comment so nobody optimises it again.

The wake word got the same treatment. The performance log showed 7 of 18 attempts matching; the other 11 had decoded as "none", "Nun", "Nahn", "Null" and "nerd", scoring between 0.57 and 0.75 against the target. That range proves no threshold value could have fixed it, which is why the fix was a two-tier name table rather than a tuned number.