1 Commits
Author SHA1 Message Date
kato 55fbd357b0 docs: update README with VAD, new threshold option, and silence handling
Release / Tests before release (push) Successful in 1m29s
Release / Build and publish release (push) Successful in 3m5s
Tests / Go and GTK tests (push) Successful in 1m39s
Update README to clarify VAD mode behavior: provisional caption only after Silero detects speech, Parakeet idle during silence. Document new --vad-threshold option to control Silero speech confidence. Add troubleshooting tip for music/noise false detections.
2026-07-17 10:35:59 +03:00
8 changed files with 226 additions and 41 deletions
+10 -2
View File
@@ -23,7 +23,7 @@ This creates `models/parakeet-tdt-v2/` and `models/silero_vad_v5.onnx`. They are
## Terminal captions
VAD mode is recommended for normal use. It redraws a provisional caption while speech is active and commits a timestamped caption after the utterance ends:
VAD mode is recommended for normal use. It redraws a provisional caption only after Silero detects speech and commits a timestamped caption after the utterance ends:
```bash
go run ./src/cmd/captioneer --mode=vad
@@ -74,6 +74,8 @@ Use `--output=both` to keep terminal output as well:
In VAD mode, a 500 ms pause finalizes the current speech segment. Captioneer splits that transcript again at sentence punctuation and at word-safe boundaries of at most 64 characters, so continuous speech does not become one enormous paragraph. The live provisional view keeps the latest three digestible chunks and grows vertically without ellipsizing them. Completed chunks remain visible for at least three seconds, receive extra time based on their word count, and gradually fade from white to a restrained gray as they age. When several chunks finalize together, their removal is staggered in reading order instead of deleting the whole block at once. Up to six completed chunks remain above the provisional text; if a seventh arrives, the oldest is removed. Blank recognition results never erase completed text before its reading lifetime expires.
The expensive Parakeet recognizer stays idle during silence and audio that Silero does not classify as speech. A small rolling preview window also prevents long continuous audio from making each refresh progressively slower. Silero itself continues receiving the live stream because it needs the quiet frames to detect when speech ends and when it starts again.
Text is left-aligned inside the centered, fixed-width dark bubble and wraps at its configured width instead of extending off-screen. Pointer input passes through the window by default.
To build outside the helper script, install GTK4 and GTK4 Layer Shell development packages, then run:
@@ -105,7 +107,8 @@ Layer Shell and X11 hints are requests, not universal guarantees. Same-layer win
| `--chunk-duration=1s` | `1s` | Fixed chunk length or VAD preview refresh interval. Accepts Go durations such as `500ms` or `2s`. |
| `--models-dir=models` | `models` | Directory containing `parakeet-tdt-v2/` and `silero_vad_v5.onnx`. |
| `--threads=2` | `2` | CPU threads used by recognition and VAD. |
| `--preview-threshold-dbfs=-45` | `-45` | RMS threshold that starts a provisional VAD caption. Raise it to reject background audio; lower it for quiet speech. |
| `--preview-threshold-dbfs=-45` | `-45` | RMS gate below which recognition stays idle. Raise it to reject quiet background noise; lower it for quiet speech. In fixed mode, gated chunks are not sent to Parakeet. |
| `--vad-threshold=0.5` | `0.5` | Silero speech confidence from `0` to `1`. Raise it to reject more music/noise; lower it if real speech is missed. Applies to VAD mode. |
| `--model-auto-restart=true` | `true` | Restart the isolated model worker after a crash, capture failure, or timeout. Set `false` to report the failure and exit instead. |
| `--model-timeout=30s` | `30s` | Maximum model startup, continuously busy processing time, or missing-heartbeat period before the worker is considered hung. `0s` disables hang detection. |
| `--model-shutdown-timeout=2s` | `2s` | Time allowed for pending-audio flush and model cleanup after Ctrl+C or reload before the worker is force-killed. |
@@ -138,6 +141,10 @@ Examples:
# Fixed two-second captions on a named monitor
./scripts/distrobox/run.sh --mode=fixed --chunk-duration=2s \
--output=overlay --overlay-monitor=DP-1
# Be more conservative when instrumental music causes false speech detections
./scripts/distrobox/run.sh --mode=vad --output=overlay \
--vad-threshold=0.65 --preview-threshold-dbfs=-40
```
## Bazzite development with Distrobox
@@ -173,6 +180,7 @@ Captioneer itself has no environment-based application configuration; use the co
- **Missing model files:** run `./scripts/download-models.sh` or correct `--models-dir`.
- **`pactl` or `parec` missing:** install PulseAudio utilities, or use the Distrobox scripts.
- **No captions:** confirm the current sink with `pactl get-default-sink`; Captioneer listens to `<sink>.monitor`.
- **Music/noise produces captions:** use VAD mode and try `--vad-threshold=0.65`. Increase it in small steps if false detections continue. `--preview-threshold-dbfs=-40` can also reject quiet background audio, but an RMS gate cannot distinguish loud music from speech. Singing may still be treated as speech.
- **Processing overload warning:** inference fell behind and the oldest 100 ms packet was dropped. Try more `--threads` or a longer `--chunk-duration`.
- **Overlay is not always above other windows:** read the backend warning. On Wayland this usually means the compositor lacks Layer Shell; add a compositor rule for the `Captioneer Overlay` window if supported.
- **Overlay initialization fails:** `--output=overlay` exits with the error. `--output=both` warns and continues terminal-only.
+1
View File
@@ -48,6 +48,7 @@ func TestWorkerSettingsRoundTrip(t *testing.T) {
ModelsDir: "models with spaces",
Threads: 3,
PreviewThresholdDB: -42,
VADThreshold: 0.65,
ModelAutoRestart: true,
ModelTimeout: 12 * time.Second,
ModelShutdownTimeout: 2 * time.Second,
+1
View File
@@ -20,6 +20,7 @@ type Event struct {
type Transcriber interface {
Decode(samples []float32) string
AcceptVAD(samples []float32)
SpeechActive() bool
FlushVAD()
NextSpeechSegment() (start int, samples []float32, ok bool)
}
+57 -7
View File
@@ -3,11 +3,18 @@ package captions
import (
"strings"
"time"
"tea.chunkbyte.com/kato/captioneer/src/audio"
"tea.chunkbyte.com/kato/captioneer/src/config"
)
// Provisional recognition uses only a recent window. Re-decoding an entire
// long utterance every refresh grows increasingly expensive, while the UI only
// presents the latest provisional lines. The final event still decodes the
// complete VAD segment.
const maxPreviewDuration = 10 * time.Second
type Processor struct {
settings config.Settings
transcriber Transcriber
@@ -18,6 +25,7 @@ type Processor struct {
previewActive bool
previewStart int
nextPreviewAt int
previewReceived int
totalSamples int
}
@@ -43,19 +51,26 @@ func (p *Processor) Accept(samples []float32) error {
func (p *Processor) Flush() error {
if p.settings.Mode == config.ModeFixed {
if len(p.fixedSamples) > 0 {
return p.emitFinal(p.totalSamples, p.totalSamples+len(p.fixedSamples), p.fixedSamples)
return p.emitFixedFinal(p.totalSamples, p.totalSamples+len(p.fixedSamples), p.fixedSamples)
}
return nil
}
p.transcriber.FlushVAD()
return p.drainVAD()
if err := p.drainVAD(); err != nil {
return err
}
if p.previewActive {
p.resetPreview()
return p.emit(Event{Kind: Hide})
}
return nil
}
func (p *Processor) acceptFixed(samples []float32) error {
p.fixedSamples = append(p.fixedSamples, samples...)
chunkSize := audio.DurationSamples(p.settings.ChunkDuration)
for len(p.fixedSamples) >= chunkSize {
if err := p.emitFinal(p.totalSamples, p.totalSamples+chunkSize, p.fixedSamples[:chunkSize]); err != nil {
if err := p.emitFixedFinal(p.totalSamples, p.totalSamples+chunkSize, p.fixedSamples[:chunkSize]); err != nil {
return err
}
p.fixedSamples = p.fixedSamples[chunkSize:]
@@ -68,20 +83,33 @@ func (p *Processor) acceptVAD(samples []float32) error {
p.totalSamples += len(samples)
p.transcriber.AcceptVAD(samples)
if p.previewActive || audio.RMSDBFS(samples) >= p.settings.PreviewThresholdDB {
speechActive := p.transcriber.SpeechActive()
if speechActive && (p.previewActive || audio.RMSDBFS(samples) >= p.settings.PreviewThresholdDB) {
if !p.previewActive {
p.previewActive = true
p.previewStart = p.totalSamples - len(samples)
}
p.previewSamples = append(p.previewSamples, samples...)
if len(p.previewSamples) >= p.nextPreviewAt {
p.previewReceived += len(samples)
p.trimPreviewWindow()
if p.previewReceived >= p.nextPreviewAt {
if err := p.emitProvisional(); err != nil {
return err
}
p.nextPreviewAt += audio.DurationSamples(p.settings.ChunkDuration)
}
}
return p.drainVAD()
if err := p.drainVAD(); err != nil {
return err
}
// A short VAD false-positive can become active without meeting the minimum
// speech duration, so no final segment is produced. Clear its provisional
// text instead of leaving a stale caption on screen.
if !speechActive && p.previewActive {
p.resetPreview()
return p.emit(Event{Kind: Hide})
}
return nil
}
func (p *Processor) drainVAD() error {
@@ -93,11 +121,26 @@ func (p *Processor) drainVAD() error {
if err := p.emitFinal(start, start+len(samples), samples); err != nil {
return err
}
p.resetPreview()
}
}
func (p *Processor) trimPreviewWindow() {
maxSamples := audio.DurationSamples(maxPreviewDuration)
if len(p.previewSamples) <= maxSamples {
return
}
dropped := len(p.previewSamples) - maxSamples
p.previewSamples = p.previewSamples[dropped:]
p.previewStart += dropped
}
func (p *Processor) resetPreview() {
p.previewSamples = nil
p.previewActive = false
p.previewStart = 0
p.previewReceived = 0
p.nextPreviewAt = audio.DurationSamples(p.settings.ChunkDuration)
}
}
func (p *Processor) emitProvisional() error {
@@ -125,3 +168,10 @@ func (p *Processor) emitFinal(start, end int, samples []float32) error {
EndedAt: audio.SamplesDuration(end),
})
}
func (p *Processor) emitFixedFinal(start, end int, samples []float32) error {
if audio.RMSDBFS(samples) < p.settings.PreviewThresholdDB {
return p.emit(Event{Kind: Hide})
}
return p.emitFinal(start, end, samples)
}
+125 -13
View File
@@ -18,9 +18,13 @@ type fakeTranscriber struct {
decodeTexts []string
segments []fakeSegment
flushed bool
speechActive bool
decodeSizes []int
acceptedVAD int
}
func (f *fakeTranscriber) Decode([]float32) string {
func (f *fakeTranscriber) Decode(samples []float32) string {
f.decodeSizes = append(f.decodeSizes, len(samples))
if len(f.decodeTexts) == 0 {
return f.text
}
@@ -28,7 +32,8 @@ func (f *fakeTranscriber) Decode([]float32) string {
f.decodeTexts = f.decodeTexts[1:]
return text
}
func (f *fakeTranscriber) AcceptVAD([]float32) {}
func (f *fakeTranscriber) AcceptVAD(samples []float32) { f.acceptedVAD += len(samples) }
func (f *fakeTranscriber) SpeechActive() bool { return f.speechActive }
func (f *fakeTranscriber) FlushVAD() { f.flushed = true }
func (f *fakeTranscriber) NextSpeechSegment() (int, []float32, bool) {
if len(f.segments) == 0 {
@@ -42,12 +47,14 @@ func (f *fakeTranscriber) NextSpeechSegment() (int, []float32, bool) {
func TestFixedModeEmitsFinalCaption(t *testing.T) {
transcriber := &fakeTranscriber{text: "hello"}
var events []Event
processor := New(config.Settings{Mode: config.ModeFixed, ChunkDuration: time.Second}, transcriber, func(event Event) error {
processor := New(config.Settings{
Mode: config.ModeFixed, ChunkDuration: time.Second, PreviewThresholdDB: -45,
}, transcriber, func(event Event) error {
events = append(events, event)
return nil
})
if err := processor.Accept(make([]float32, audio.SampleRate)); err != nil {
if err := processor.Accept(constantSamples(audio.SampleRate, 0.5)); err != nil {
t.Fatal(err)
}
if len(events) != 1 || events[0].Kind != Final || events[0].Text != "hello" {
@@ -61,12 +68,14 @@ func TestFixedModeEmitsFinalCaption(t *testing.T) {
func TestFixedModeFlushesFinalPartialChunkWithExactTimestamps(t *testing.T) {
transcriber := &fakeTranscriber{decodeTexts: []string{"first", "partial"}}
var events []Event
processor := New(config.Settings{Mode: config.ModeFixed, ChunkDuration: time.Second}, transcriber, func(event Event) error {
processor := New(config.Settings{
Mode: config.ModeFixed, ChunkDuration: time.Second, PreviewThresholdDB: -45,
}, transcriber, func(event Event) error {
events = append(events, event)
return nil
})
if err := processor.Accept(make([]float32, audio.SampleRate+audio.SampleRate/2)); err != nil {
if err := processor.Accept(constantSamples(audio.SampleRate+audio.SampleRate/2, 0.5)); err != nil {
t.Fatal(err)
}
if err := processor.Flush(); err != nil {
@@ -81,7 +90,7 @@ func TestFixedModeFlushesFinalPartialChunkWithExactTimestamps(t *testing.T) {
}
func TestVADModeEmitsProvisionalThenFinal(t *testing.T) {
transcriber := &fakeTranscriber{text: "speech"}
transcriber := &fakeTranscriber{text: "speech", speechActive: true}
var events []Event
processor := New(config.Settings{
Mode: config.ModeVAD,
@@ -92,14 +101,12 @@ func TestVADModeEmitsProvisionalThenFinal(t *testing.T) {
return nil
})
loud := make([]float32, audio.SampleRate)
for i := range loud {
loud[i] = 0.5
}
loud := constantSamples(audio.SampleRate, 0.5)
if err := processor.Accept(loud); err != nil {
t.Fatal(err)
}
transcriber.segments = []fakeSegment{{start: 0, samples: loud}}
transcriber.speechActive = false
if err := processor.Accept(make([]float32, audio.SamplesPerPacket())); err != nil {
t.Fatal(err)
}
@@ -112,11 +119,13 @@ func TestVADModeEmitsProvisionalThenFinal(t *testing.T) {
func TestEmptyFinalEmitsHide(t *testing.T) {
transcriber := &fakeTranscriber{text: " \n\t "}
var event Event
processor := New(config.Settings{Mode: config.ModeFixed, ChunkDuration: time.Second}, transcriber, func(got Event) error {
processor := New(config.Settings{
Mode: config.ModeFixed, ChunkDuration: time.Second, PreviewThresholdDB: -45,
}, transcriber, func(got Event) error {
event = got
return nil
})
if err := processor.Accept(make([]float32, audio.SampleRate)); err != nil {
if err := processor.Accept(constantSamples(audio.SampleRate, 0.5)); err != nil {
t.Fatal(err)
}
if event.Kind != Hide {
@@ -124,6 +133,101 @@ func TestEmptyFinalEmitsHide(t *testing.T) {
}
}
func TestFixedModeSilenceStaysOutOfRecognizer(t *testing.T) {
transcriber := &fakeTranscriber{text: "hallucinated caption"}
var events []Event
processor := New(config.Settings{
Mode: config.ModeFixed,
ChunkDuration: time.Second,
PreviewThresholdDB: -45,
}, transcriber, func(event Event) error {
events = append(events, event)
return nil
})
if err := processor.Accept(make([]float32, audio.SampleRate)); err != nil {
t.Fatal(err)
}
if len(transcriber.decodeSizes) != 0 {
t.Fatalf("silent audio reached recognizer: decode sizes = %v", transcriber.decodeSizes)
}
if len(events) != 1 || events[0].Kind != Hide {
t.Fatalf("silent chunk events = %#v, want one Hide", events)
}
}
func TestVADModeDoesNotPreviewLoudNonSpeech(t *testing.T) {
transcriber := &fakeTranscriber{text: "music hallucination"}
var events []Event
processor := New(config.Settings{
Mode: config.ModeVAD,
ChunkDuration: time.Second,
PreviewThresholdDB: -45,
}, transcriber, func(event Event) error {
events = append(events, event)
return nil
})
loudMusic := constantSamples(audio.SampleRate, 0.5)
if err := processor.Accept(loudMusic); err != nil {
t.Fatal(err)
}
if transcriber.acceptedVAD != len(loudMusic) {
t.Fatalf("VAD received %d samples, want %d", transcriber.acceptedVAD, len(loudMusic))
}
if len(transcriber.decodeSizes) != 0 || len(events) != 0 {
t.Fatalf("non-speech reached recognizer: decodes=%v events=%#v", transcriber.decodeSizes, events)
}
}
func TestVADModeClearsRejectedShortSpeechPreview(t *testing.T) {
transcriber := &fakeTranscriber{text: "draft", speechActive: true}
var events []Event
processor := New(config.Settings{
Mode: config.ModeVAD,
ChunkDuration: time.Second,
PreviewThresholdDB: -45,
}, transcriber, func(event Event) error {
events = append(events, event)
return nil
})
if err := processor.Accept(constantSamples(audio.SampleRate, 0.5)); err != nil {
t.Fatal(err)
}
transcriber.speechActive = false
if err := processor.Accept(make([]float32, audio.SamplesPerPacket())); err != nil {
t.Fatal(err)
}
if len(events) != 2 || events[0].Kind != Provisional || events[1].Kind != Hide {
t.Fatalf("events = %#v, want Provisional then Hide", events)
}
}
func TestVADPreviewRecognitionWindowIsBounded(t *testing.T) {
transcriber := &fakeTranscriber{text: "draft", speechActive: true}
processor := New(config.Settings{
Mode: config.ModeVAD,
ChunkDuration: time.Second,
PreviewThresholdDB: -45,
}, transcriber, func(Event) error { return nil })
for range 15 {
if err := processor.Accept(constantSamples(audio.SampleRate, 0.5)); err != nil {
t.Fatal(err)
}
}
maxSamples := audio.DurationSamples(maxPreviewDuration)
for _, size := range transcriber.decodeSizes {
if size > maxSamples {
t.Fatalf("preview decode used %d samples, limit is %d", size, maxSamples)
}
}
if len(transcriber.decodeSizes) != 15 {
t.Fatalf("preview decode count = %d, want 15", len(transcriber.decodeSizes))
}
}
func TestFlushFlushesVAD(t *testing.T) {
transcriber := &fakeTranscriber{}
processor := New(config.Settings{Mode: config.ModeVAD, ChunkDuration: time.Second}, transcriber, func(Event) error { return nil })
@@ -134,3 +238,11 @@ func TestFlushFlushesVAD(t *testing.T) {
t.Fatal("VAD was not flushed")
}
}
func constantSamples(count int, value float32) []float32 {
samples := make([]float32, count)
for i := range samples {
samples[i] = value
}
return samples
}
+9 -2
View File
@@ -10,6 +10,7 @@ import (
const (
DefaultModelsDir = "models"
DefaultVADThreshold = 0.5
DefaultOverlayFontFamily = "Sans"
MinimumOverlayFinalLifetime = 3 * time.Second
DefaultModelTimeout = 30 * time.Second
@@ -29,6 +30,7 @@ type Settings struct {
ModelsDir string
Threads int
PreviewThresholdDB float64
VADThreshold float64
ModelAutoRestart bool
ModelTimeout time.Duration
ModelShutdownTimeout time.Duration
@@ -41,7 +43,8 @@ func Parse() Settings {
flag.DurationVar(&settings.ChunkDuration, "chunk-duration", time.Second, "fixed chunk size or VAD preview refresh interval")
flag.StringVar(&settings.ModelsDir, "models-dir", DefaultModelsDir, "directory containing downloaded Sherpa models")
flag.IntVar(&settings.Threads, "threads", 2, "CPU threads used by recognition and VAD")
flag.Float64Var(&settings.PreviewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS threshold used to begin VAD previews")
flag.Float64Var(&settings.PreviewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS gate below which recognition stays idle")
flag.Float64Var(&settings.VADThreshold, "vad-threshold", DefaultVADThreshold, "speech detection sensitivity from 0 to 1; higher rejects more non-speech")
flag.BoolVar(&settings.ModelAutoRestart, "model-auto-restart", true, "restart the isolated model worker after a crash or timeout")
flag.DurationVar(&settings.ModelTimeout, "model-timeout", DefaultModelTimeout, "maximum model startup or processing time; zero disables hang detection")
flag.DurationVar(&settings.ModelShutdownTimeout, "model-shutdown-timeout", DefaultModelShutdownTimeout, "time allowed for model flush and cleanup before force-killing the worker")
@@ -94,7 +97,8 @@ func ParseDesktop() DesktopSettings {
flags.DurationVar(&desktop.ChunkDuration, "chunk-duration", time.Second, "fixed chunk size or VAD preview refresh interval")
flags.StringVar(&desktop.ModelsDir, "models-dir", DefaultModelsDir, "directory containing downloaded Sherpa models")
flags.IntVar(&desktop.Threads, "threads", 2, "CPU threads used by recognition and VAD")
flags.Float64Var(&desktop.PreviewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS threshold used to begin VAD previews")
flags.Float64Var(&desktop.PreviewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS gate below which recognition stays idle")
flags.Float64Var(&desktop.VADThreshold, "vad-threshold", DefaultVADThreshold, "speech detection sensitivity from 0 to 1; higher rejects more non-speech")
flags.BoolVar(&desktop.ModelAutoRestart, "model-auto-restart", true, "restart the isolated model worker after a crash or timeout")
flags.DurationVar(&desktop.ModelTimeout, "model-timeout", DefaultModelTimeout, "maximum model startup or processing time; zero disables hang detection")
flags.DurationVar(&desktop.ModelShutdownTimeout, "model-shutdown-timeout", DefaultModelShutdownTimeout, "time allowed for model flush and cleanup before force-killing the worker")
@@ -125,6 +129,9 @@ func (s Settings) Validate() error {
if s.Threads <= 0 {
return errors.New("--threads must be positive")
}
if s.VADThreshold < 0 || s.VADThreshold > 1 {
return errors.New("--vad-threshold must be between 0 and 1")
}
if s.ModelTimeout < 0 {
return errors.New("--model-timeout cannot be negative")
}
+2
View File
@@ -59,6 +59,8 @@ func TestDesktopSettingsValidate(t *testing.T) {
{"short timeout", func(s *DesktopSettings) { s.Overlay.FinalTimeout = time.Second }},
{"negative model timeout", func(s *DesktopSettings) { s.ModelTimeout = -time.Second }},
{"zero model shutdown timeout", func(s *DesktopSettings) { s.ModelShutdownTimeout = 0 }},
{"negative VAD threshold", func(s *DesktopSettings) { s.VADThreshold = -0.1 }},
{"large VAD threshold", func(s *DesktopSettings) { s.VADThreshold = 1.1 }},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
+7 -3
View File
@@ -57,7 +57,7 @@ func New(settings config.Settings) (*Resources, error) {
resources := &Resources{recognizer: recognizer}
if settings.Mode == config.ModeVAD {
resources.vad = newVAD(paths.vad, settings.Threads)
resources.vad = newVAD(paths.vad, settings.Threads, settings.VADThreshold)
if resources.vad == nil {
resources.Close()
return nil, fmt.Errorf("create Silero VAD: Sherpa returned nil")
@@ -92,6 +92,10 @@ func (r *Resources) AcceptVAD(samples []float32) {
r.vad.AcceptWaveform(samples)
}
func (r *Resources) SpeechActive() bool {
return r.vad.IsSpeech()
}
func (r *Resources) FlushVAD() {
r.vad.Flush()
}
@@ -132,11 +136,11 @@ func validatePaths(modelsDir string, mode config.Mode) (modelPaths, error) {
return paths, nil
}
func newVAD(model string, threads int) *sherpa.VoiceActivityDetector {
func newVAD(model string, threads int, threshold float64) *sherpa.VoiceActivityDetector {
return sherpa.NewVoiceActivityDetector(&sherpa.VadModelConfig{
SileroVad: sherpa.SileroVadModelConfig{
Model: model,
Threshold: 0.5,
Threshold: float32(threshold),
// A 500 ms pause ends the current caption line and starts the next one.
MinSilenceDuration: 0.5,
MinSpeechDuration: 0.25,