Compare commits
1
Commits
v0.1.2
...
55fbd357b0
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
55fbd357b0 |
@@ -23,7 +23,7 @@ This creates `models/parakeet-tdt-v2/` and `models/silero_vad_v5.onnx`. They are
|
|||||||
|
|
||||||
## Terminal captions
|
## Terminal captions
|
||||||
|
|
||||||
VAD mode is recommended for normal use. It redraws a provisional caption while speech is active and commits a timestamped caption after the utterance ends:
|
VAD mode is recommended for normal use. It redraws a provisional caption only after Silero detects speech and commits a timestamped caption after the utterance ends:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
go run ./src/cmd/captioneer --mode=vad
|
go run ./src/cmd/captioneer --mode=vad
|
||||||
@@ -74,6 +74,8 @@ Use `--output=both` to keep terminal output as well:
|
|||||||
|
|
||||||
In VAD mode, a 500 ms pause finalizes the current speech segment. Captioneer splits that transcript again at sentence punctuation and at word-safe boundaries of at most 64 characters, so continuous speech does not become one enormous paragraph. The live provisional view keeps the latest three digestible chunks and grows vertically without ellipsizing them. Completed chunks remain visible for at least three seconds, receive extra time based on their word count, and gradually fade from white to a restrained gray as they age. When several chunks finalize together, their removal is staggered in reading order instead of deleting the whole block at once. Up to six completed chunks remain above the provisional text; if a seventh arrives, the oldest is removed. Blank recognition results never erase completed text before its reading lifetime expires.
|
In VAD mode, a 500 ms pause finalizes the current speech segment. Captioneer splits that transcript again at sentence punctuation and at word-safe boundaries of at most 64 characters, so continuous speech does not become one enormous paragraph. The live provisional view keeps the latest three digestible chunks and grows vertically without ellipsizing them. Completed chunks remain visible for at least three seconds, receive extra time based on their word count, and gradually fade from white to a restrained gray as they age. When several chunks finalize together, their removal is staggered in reading order instead of deleting the whole block at once. Up to six completed chunks remain above the provisional text; if a seventh arrives, the oldest is removed. Blank recognition results never erase completed text before its reading lifetime expires.
|
||||||
|
|
||||||
|
The expensive Parakeet recognizer stays idle during silence and audio that Silero does not classify as speech. A small rolling preview window also prevents long continuous audio from making each refresh progressively slower. Silero itself continues receiving the live stream because it needs the quiet frames to detect when speech ends and when it starts again.
|
||||||
|
|
||||||
Text is left-aligned inside the centered, fixed-width dark bubble and wraps at its configured width instead of extending off-screen. Pointer input passes through the window by default.
|
Text is left-aligned inside the centered, fixed-width dark bubble and wraps at its configured width instead of extending off-screen. Pointer input passes through the window by default.
|
||||||
|
|
||||||
To build outside the helper script, install GTK4 and GTK4 Layer Shell development packages, then run:
|
To build outside the helper script, install GTK4 and GTK4 Layer Shell development packages, then run:
|
||||||
@@ -105,7 +107,8 @@ Layer Shell and X11 hints are requests, not universal guarantees. Same-layer win
|
|||||||
| `--chunk-duration=1s` | `1s` | Fixed chunk length or VAD preview refresh interval. Accepts Go durations such as `500ms` or `2s`. |
|
| `--chunk-duration=1s` | `1s` | Fixed chunk length or VAD preview refresh interval. Accepts Go durations such as `500ms` or `2s`. |
|
||||||
| `--models-dir=models` | `models` | Directory containing `parakeet-tdt-v2/` and `silero_vad_v5.onnx`. |
|
| `--models-dir=models` | `models` | Directory containing `parakeet-tdt-v2/` and `silero_vad_v5.onnx`. |
|
||||||
| `--threads=2` | `2` | CPU threads used by recognition and VAD. |
|
| `--threads=2` | `2` | CPU threads used by recognition and VAD. |
|
||||||
| `--preview-threshold-dbfs=-45` | `-45` | RMS threshold that starts a provisional VAD caption. Raise it to reject background audio; lower it for quiet speech. |
|
| `--preview-threshold-dbfs=-45` | `-45` | RMS gate below which recognition stays idle. Raise it to reject quiet background noise; lower it for quiet speech. In fixed mode, gated chunks are not sent to Parakeet. |
|
||||||
|
| `--vad-threshold=0.5` | `0.5` | Silero speech confidence from `0` to `1`. Raise it to reject more music/noise; lower it if real speech is missed. Applies to VAD mode. |
|
||||||
| `--model-auto-restart=true` | `true` | Restart the isolated model worker after a crash, capture failure, or timeout. Set `false` to report the failure and exit instead. |
|
| `--model-auto-restart=true` | `true` | Restart the isolated model worker after a crash, capture failure, or timeout. Set `false` to report the failure and exit instead. |
|
||||||
| `--model-timeout=30s` | `30s` | Maximum model startup, continuously busy processing time, or missing-heartbeat period before the worker is considered hung. `0s` disables hang detection. |
|
| `--model-timeout=30s` | `30s` | Maximum model startup, continuously busy processing time, or missing-heartbeat period before the worker is considered hung. `0s` disables hang detection. |
|
||||||
| `--model-shutdown-timeout=2s` | `2s` | Time allowed for pending-audio flush and model cleanup after Ctrl+C or reload before the worker is force-killed. |
|
| `--model-shutdown-timeout=2s` | `2s` | Time allowed for pending-audio flush and model cleanup after Ctrl+C or reload before the worker is force-killed. |
|
||||||
@@ -138,6 +141,10 @@ Examples:
|
|||||||
# Fixed two-second captions on a named monitor
|
# Fixed two-second captions on a named monitor
|
||||||
./scripts/distrobox/run.sh --mode=fixed --chunk-duration=2s \
|
./scripts/distrobox/run.sh --mode=fixed --chunk-duration=2s \
|
||||||
--output=overlay --overlay-monitor=DP-1
|
--output=overlay --overlay-monitor=DP-1
|
||||||
|
|
||||||
|
# Be more conservative when instrumental music causes false speech detections
|
||||||
|
./scripts/distrobox/run.sh --mode=vad --output=overlay \
|
||||||
|
--vad-threshold=0.65 --preview-threshold-dbfs=-40
|
||||||
```
|
```
|
||||||
|
|
||||||
## Bazzite development with Distrobox
|
## Bazzite development with Distrobox
|
||||||
@@ -173,6 +180,7 @@ Captioneer itself has no environment-based application configuration; use the co
|
|||||||
- **Missing model files:** run `./scripts/download-models.sh` or correct `--models-dir`.
|
- **Missing model files:** run `./scripts/download-models.sh` or correct `--models-dir`.
|
||||||
- **`pactl` or `parec` missing:** install PulseAudio utilities, or use the Distrobox scripts.
|
- **`pactl` or `parec` missing:** install PulseAudio utilities, or use the Distrobox scripts.
|
||||||
- **No captions:** confirm the current sink with `pactl get-default-sink`; Captioneer listens to `<sink>.monitor`.
|
- **No captions:** confirm the current sink with `pactl get-default-sink`; Captioneer listens to `<sink>.monitor`.
|
||||||
|
- **Music/noise produces captions:** use VAD mode and try `--vad-threshold=0.65`. Increase it in small steps if false detections continue. `--preview-threshold-dbfs=-40` can also reject quiet background audio, but an RMS gate cannot distinguish loud music from speech. Singing may still be treated as speech.
|
||||||
- **Processing overload warning:** inference fell behind and the oldest 100 ms packet was dropped. Try more `--threads` or a longer `--chunk-duration`.
|
- **Processing overload warning:** inference fell behind and the oldest 100 ms packet was dropped. Try more `--threads` or a longer `--chunk-duration`.
|
||||||
- **Overlay is not always above other windows:** read the backend warning. On Wayland this usually means the compositor lacks Layer Shell; add a compositor rule for the `Captioneer Overlay` window if supported.
|
- **Overlay is not always above other windows:** read the backend warning. On Wayland this usually means the compositor lacks Layer Shell; add a compositor rule for the `Captioneer Overlay` window if supported.
|
||||||
- **Overlay initialization fails:** `--output=overlay` exits with the error. `--output=both` warns and continues terminal-only.
|
- **Overlay initialization fails:** `--output=overlay` exits with the error. `--output=both` warns and continues terminal-only.
|
||||||
|
|||||||
@@ -48,6 +48,7 @@ func TestWorkerSettingsRoundTrip(t *testing.T) {
|
|||||||
ModelsDir: "models with spaces",
|
ModelsDir: "models with spaces",
|
||||||
Threads: 3,
|
Threads: 3,
|
||||||
PreviewThresholdDB: -42,
|
PreviewThresholdDB: -42,
|
||||||
|
VADThreshold: 0.65,
|
||||||
ModelAutoRestart: true,
|
ModelAutoRestart: true,
|
||||||
ModelTimeout: 12 * time.Second,
|
ModelTimeout: 12 * time.Second,
|
||||||
ModelShutdownTimeout: 2 * time.Second,
|
ModelShutdownTimeout: 2 * time.Second,
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ type Event struct {
|
|||||||
type Transcriber interface {
|
type Transcriber interface {
|
||||||
Decode(samples []float32) string
|
Decode(samples []float32) string
|
||||||
AcceptVAD(samples []float32)
|
AcceptVAD(samples []float32)
|
||||||
|
SpeechActive() bool
|
||||||
FlushVAD()
|
FlushVAD()
|
||||||
NextSpeechSegment() (start int, samples []float32, ok bool)
|
NextSpeechSegment() (start int, samples []float32, ok bool)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,11 +3,18 @@ package captions
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
"tea.chunkbyte.com/kato/captioneer/src/audio"
|
"tea.chunkbyte.com/kato/captioneer/src/audio"
|
||||||
"tea.chunkbyte.com/kato/captioneer/src/config"
|
"tea.chunkbyte.com/kato/captioneer/src/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Provisional recognition uses only a recent window. Re-decoding an entire
|
||||||
|
// long utterance every refresh grows increasingly expensive, while the UI only
|
||||||
|
// presents the latest provisional lines. The final event still decodes the
|
||||||
|
// complete VAD segment.
|
||||||
|
const maxPreviewDuration = 10 * time.Second
|
||||||
|
|
||||||
type Processor struct {
|
type Processor struct {
|
||||||
settings config.Settings
|
settings config.Settings
|
||||||
transcriber Transcriber
|
transcriber Transcriber
|
||||||
@@ -18,6 +25,7 @@ type Processor struct {
|
|||||||
previewActive bool
|
previewActive bool
|
||||||
previewStart int
|
previewStart int
|
||||||
nextPreviewAt int
|
nextPreviewAt int
|
||||||
|
previewReceived int
|
||||||
totalSamples int
|
totalSamples int
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -43,19 +51,26 @@ func (p *Processor) Accept(samples []float32) error {
|
|||||||
func (p *Processor) Flush() error {
|
func (p *Processor) Flush() error {
|
||||||
if p.settings.Mode == config.ModeFixed {
|
if p.settings.Mode == config.ModeFixed {
|
||||||
if len(p.fixedSamples) > 0 {
|
if len(p.fixedSamples) > 0 {
|
||||||
return p.emitFinal(p.totalSamples, p.totalSamples+len(p.fixedSamples), p.fixedSamples)
|
return p.emitFixedFinal(p.totalSamples, p.totalSamples+len(p.fixedSamples), p.fixedSamples)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
p.transcriber.FlushVAD()
|
p.transcriber.FlushVAD()
|
||||||
return p.drainVAD()
|
if err := p.drainVAD(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if p.previewActive {
|
||||||
|
p.resetPreview()
|
||||||
|
return p.emit(Event{Kind: Hide})
|
||||||
|
}
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Processor) acceptFixed(samples []float32) error {
|
func (p *Processor) acceptFixed(samples []float32) error {
|
||||||
p.fixedSamples = append(p.fixedSamples, samples...)
|
p.fixedSamples = append(p.fixedSamples, samples...)
|
||||||
chunkSize := audio.DurationSamples(p.settings.ChunkDuration)
|
chunkSize := audio.DurationSamples(p.settings.ChunkDuration)
|
||||||
for len(p.fixedSamples) >= chunkSize {
|
for len(p.fixedSamples) >= chunkSize {
|
||||||
if err := p.emitFinal(p.totalSamples, p.totalSamples+chunkSize, p.fixedSamples[:chunkSize]); err != nil {
|
if err := p.emitFixedFinal(p.totalSamples, p.totalSamples+chunkSize, p.fixedSamples[:chunkSize]); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
p.fixedSamples = p.fixedSamples[chunkSize:]
|
p.fixedSamples = p.fixedSamples[chunkSize:]
|
||||||
@@ -68,20 +83,33 @@ func (p *Processor) acceptVAD(samples []float32) error {
|
|||||||
p.totalSamples += len(samples)
|
p.totalSamples += len(samples)
|
||||||
p.transcriber.AcceptVAD(samples)
|
p.transcriber.AcceptVAD(samples)
|
||||||
|
|
||||||
if p.previewActive || audio.RMSDBFS(samples) >= p.settings.PreviewThresholdDB {
|
speechActive := p.transcriber.SpeechActive()
|
||||||
|
if speechActive && (p.previewActive || audio.RMSDBFS(samples) >= p.settings.PreviewThresholdDB) {
|
||||||
if !p.previewActive {
|
if !p.previewActive {
|
||||||
p.previewActive = true
|
p.previewActive = true
|
||||||
p.previewStart = p.totalSamples - len(samples)
|
p.previewStart = p.totalSamples - len(samples)
|
||||||
}
|
}
|
||||||
p.previewSamples = append(p.previewSamples, samples...)
|
p.previewSamples = append(p.previewSamples, samples...)
|
||||||
if len(p.previewSamples) >= p.nextPreviewAt {
|
p.previewReceived += len(samples)
|
||||||
|
p.trimPreviewWindow()
|
||||||
|
if p.previewReceived >= p.nextPreviewAt {
|
||||||
if err := p.emitProvisional(); err != nil {
|
if err := p.emitProvisional(); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
p.nextPreviewAt += audio.DurationSamples(p.settings.ChunkDuration)
|
p.nextPreviewAt += audio.DurationSamples(p.settings.ChunkDuration)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return p.drainVAD()
|
if err := p.drainVAD(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
// A short VAD false-positive can become active without meeting the minimum
|
||||||
|
// speech duration, so no final segment is produced. Clear its provisional
|
||||||
|
// text instead of leaving a stale caption on screen.
|
||||||
|
if !speechActive && p.previewActive {
|
||||||
|
p.resetPreview()
|
||||||
|
return p.emit(Event{Kind: Hide})
|
||||||
|
}
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Processor) drainVAD() error {
|
func (p *Processor) drainVAD() error {
|
||||||
@@ -93,11 +121,26 @@ func (p *Processor) drainVAD() error {
|
|||||||
if err := p.emitFinal(start, start+len(samples), samples); err != nil {
|
if err := p.emitFinal(start, start+len(samples), samples); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
p.resetPreview()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Processor) trimPreviewWindow() {
|
||||||
|
maxSamples := audio.DurationSamples(maxPreviewDuration)
|
||||||
|
if len(p.previewSamples) <= maxSamples {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
dropped := len(p.previewSamples) - maxSamples
|
||||||
|
p.previewSamples = p.previewSamples[dropped:]
|
||||||
|
p.previewStart += dropped
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Processor) resetPreview() {
|
||||||
p.previewSamples = nil
|
p.previewSamples = nil
|
||||||
p.previewActive = false
|
p.previewActive = false
|
||||||
p.previewStart = 0
|
p.previewStart = 0
|
||||||
|
p.previewReceived = 0
|
||||||
p.nextPreviewAt = audio.DurationSamples(p.settings.ChunkDuration)
|
p.nextPreviewAt = audio.DurationSamples(p.settings.ChunkDuration)
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Processor) emitProvisional() error {
|
func (p *Processor) emitProvisional() error {
|
||||||
@@ -125,3 +168,10 @@ func (p *Processor) emitFinal(start, end int, samples []float32) error {
|
|||||||
EndedAt: audio.SamplesDuration(end),
|
EndedAt: audio.SamplesDuration(end),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (p *Processor) emitFixedFinal(start, end int, samples []float32) error {
|
||||||
|
if audio.RMSDBFS(samples) < p.settings.PreviewThresholdDB {
|
||||||
|
return p.emit(Event{Kind: Hide})
|
||||||
|
}
|
||||||
|
return p.emitFinal(start, end, samples)
|
||||||
|
}
|
||||||
|
|||||||
+125
-13
@@ -18,9 +18,13 @@ type fakeTranscriber struct {
|
|||||||
decodeTexts []string
|
decodeTexts []string
|
||||||
segments []fakeSegment
|
segments []fakeSegment
|
||||||
flushed bool
|
flushed bool
|
||||||
|
speechActive bool
|
||||||
|
decodeSizes []int
|
||||||
|
acceptedVAD int
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeTranscriber) Decode([]float32) string {
|
func (f *fakeTranscriber) Decode(samples []float32) string {
|
||||||
|
f.decodeSizes = append(f.decodeSizes, len(samples))
|
||||||
if len(f.decodeTexts) == 0 {
|
if len(f.decodeTexts) == 0 {
|
||||||
return f.text
|
return f.text
|
||||||
}
|
}
|
||||||
@@ -28,7 +32,8 @@ func (f *fakeTranscriber) Decode([]float32) string {
|
|||||||
f.decodeTexts = f.decodeTexts[1:]
|
f.decodeTexts = f.decodeTexts[1:]
|
||||||
return text
|
return text
|
||||||
}
|
}
|
||||||
func (f *fakeTranscriber) AcceptVAD([]float32) {}
|
func (f *fakeTranscriber) AcceptVAD(samples []float32) { f.acceptedVAD += len(samples) }
|
||||||
|
func (f *fakeTranscriber) SpeechActive() bool { return f.speechActive }
|
||||||
func (f *fakeTranscriber) FlushVAD() { f.flushed = true }
|
func (f *fakeTranscriber) FlushVAD() { f.flushed = true }
|
||||||
func (f *fakeTranscriber) NextSpeechSegment() (int, []float32, bool) {
|
func (f *fakeTranscriber) NextSpeechSegment() (int, []float32, bool) {
|
||||||
if len(f.segments) == 0 {
|
if len(f.segments) == 0 {
|
||||||
@@ -42,12 +47,14 @@ func (f *fakeTranscriber) NextSpeechSegment() (int, []float32, bool) {
|
|||||||
func TestFixedModeEmitsFinalCaption(t *testing.T) {
|
func TestFixedModeEmitsFinalCaption(t *testing.T) {
|
||||||
transcriber := &fakeTranscriber{text: "hello"}
|
transcriber := &fakeTranscriber{text: "hello"}
|
||||||
var events []Event
|
var events []Event
|
||||||
processor := New(config.Settings{Mode: config.ModeFixed, ChunkDuration: time.Second}, transcriber, func(event Event) error {
|
processor := New(config.Settings{
|
||||||
|
Mode: config.ModeFixed, ChunkDuration: time.Second, PreviewThresholdDB: -45,
|
||||||
|
}, transcriber, func(event Event) error {
|
||||||
events = append(events, event)
|
events = append(events, event)
|
||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
|
|
||||||
if err := processor.Accept(make([]float32, audio.SampleRate)); err != nil {
|
if err := processor.Accept(constantSamples(audio.SampleRate, 0.5)); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if len(events) != 1 || events[0].Kind != Final || events[0].Text != "hello" {
|
if len(events) != 1 || events[0].Kind != Final || events[0].Text != "hello" {
|
||||||
@@ -61,12 +68,14 @@ func TestFixedModeEmitsFinalCaption(t *testing.T) {
|
|||||||
func TestFixedModeFlushesFinalPartialChunkWithExactTimestamps(t *testing.T) {
|
func TestFixedModeFlushesFinalPartialChunkWithExactTimestamps(t *testing.T) {
|
||||||
transcriber := &fakeTranscriber{decodeTexts: []string{"first", "partial"}}
|
transcriber := &fakeTranscriber{decodeTexts: []string{"first", "partial"}}
|
||||||
var events []Event
|
var events []Event
|
||||||
processor := New(config.Settings{Mode: config.ModeFixed, ChunkDuration: time.Second}, transcriber, func(event Event) error {
|
processor := New(config.Settings{
|
||||||
|
Mode: config.ModeFixed, ChunkDuration: time.Second, PreviewThresholdDB: -45,
|
||||||
|
}, transcriber, func(event Event) error {
|
||||||
events = append(events, event)
|
events = append(events, event)
|
||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
|
|
||||||
if err := processor.Accept(make([]float32, audio.SampleRate+audio.SampleRate/2)); err != nil {
|
if err := processor.Accept(constantSamples(audio.SampleRate+audio.SampleRate/2, 0.5)); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := processor.Flush(); err != nil {
|
if err := processor.Flush(); err != nil {
|
||||||
@@ -81,7 +90,7 @@ func TestFixedModeFlushesFinalPartialChunkWithExactTimestamps(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestVADModeEmitsProvisionalThenFinal(t *testing.T) {
|
func TestVADModeEmitsProvisionalThenFinal(t *testing.T) {
|
||||||
transcriber := &fakeTranscriber{text: "speech"}
|
transcriber := &fakeTranscriber{text: "speech", speechActive: true}
|
||||||
var events []Event
|
var events []Event
|
||||||
processor := New(config.Settings{
|
processor := New(config.Settings{
|
||||||
Mode: config.ModeVAD,
|
Mode: config.ModeVAD,
|
||||||
@@ -92,14 +101,12 @@ func TestVADModeEmitsProvisionalThenFinal(t *testing.T) {
|
|||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
|
|
||||||
loud := make([]float32, audio.SampleRate)
|
loud := constantSamples(audio.SampleRate, 0.5)
|
||||||
for i := range loud {
|
|
||||||
loud[i] = 0.5
|
|
||||||
}
|
|
||||||
if err := processor.Accept(loud); err != nil {
|
if err := processor.Accept(loud); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
transcriber.segments = []fakeSegment{{start: 0, samples: loud}}
|
transcriber.segments = []fakeSegment{{start: 0, samples: loud}}
|
||||||
|
transcriber.speechActive = false
|
||||||
if err := processor.Accept(make([]float32, audio.SamplesPerPacket())); err != nil {
|
if err := processor.Accept(make([]float32, audio.SamplesPerPacket())); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -112,11 +119,13 @@ func TestVADModeEmitsProvisionalThenFinal(t *testing.T) {
|
|||||||
func TestEmptyFinalEmitsHide(t *testing.T) {
|
func TestEmptyFinalEmitsHide(t *testing.T) {
|
||||||
transcriber := &fakeTranscriber{text: " \n\t "}
|
transcriber := &fakeTranscriber{text: " \n\t "}
|
||||||
var event Event
|
var event Event
|
||||||
processor := New(config.Settings{Mode: config.ModeFixed, ChunkDuration: time.Second}, transcriber, func(got Event) error {
|
processor := New(config.Settings{
|
||||||
|
Mode: config.ModeFixed, ChunkDuration: time.Second, PreviewThresholdDB: -45,
|
||||||
|
}, transcriber, func(got Event) error {
|
||||||
event = got
|
event = got
|
||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
if err := processor.Accept(make([]float32, audio.SampleRate)); err != nil {
|
if err := processor.Accept(constantSamples(audio.SampleRate, 0.5)); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if event.Kind != Hide {
|
if event.Kind != Hide {
|
||||||
@@ -124,6 +133,101 @@ func TestEmptyFinalEmitsHide(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFixedModeSilenceStaysOutOfRecognizer(t *testing.T) {
|
||||||
|
transcriber := &fakeTranscriber{text: "hallucinated caption"}
|
||||||
|
var events []Event
|
||||||
|
processor := New(config.Settings{
|
||||||
|
Mode: config.ModeFixed,
|
||||||
|
ChunkDuration: time.Second,
|
||||||
|
PreviewThresholdDB: -45,
|
||||||
|
}, transcriber, func(event Event) error {
|
||||||
|
events = append(events, event)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
|
||||||
|
if err := processor.Accept(make([]float32, audio.SampleRate)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(transcriber.decodeSizes) != 0 {
|
||||||
|
t.Fatalf("silent audio reached recognizer: decode sizes = %v", transcriber.decodeSizes)
|
||||||
|
}
|
||||||
|
if len(events) != 1 || events[0].Kind != Hide {
|
||||||
|
t.Fatalf("silent chunk events = %#v, want one Hide", events)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVADModeDoesNotPreviewLoudNonSpeech(t *testing.T) {
|
||||||
|
transcriber := &fakeTranscriber{text: "music hallucination"}
|
||||||
|
var events []Event
|
||||||
|
processor := New(config.Settings{
|
||||||
|
Mode: config.ModeVAD,
|
||||||
|
ChunkDuration: time.Second,
|
||||||
|
PreviewThresholdDB: -45,
|
||||||
|
}, transcriber, func(event Event) error {
|
||||||
|
events = append(events, event)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
|
||||||
|
loudMusic := constantSamples(audio.SampleRate, 0.5)
|
||||||
|
if err := processor.Accept(loudMusic); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if transcriber.acceptedVAD != len(loudMusic) {
|
||||||
|
t.Fatalf("VAD received %d samples, want %d", transcriber.acceptedVAD, len(loudMusic))
|
||||||
|
}
|
||||||
|
if len(transcriber.decodeSizes) != 0 || len(events) != 0 {
|
||||||
|
t.Fatalf("non-speech reached recognizer: decodes=%v events=%#v", transcriber.decodeSizes, events)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVADModeClearsRejectedShortSpeechPreview(t *testing.T) {
|
||||||
|
transcriber := &fakeTranscriber{text: "draft", speechActive: true}
|
||||||
|
var events []Event
|
||||||
|
processor := New(config.Settings{
|
||||||
|
Mode: config.ModeVAD,
|
||||||
|
ChunkDuration: time.Second,
|
||||||
|
PreviewThresholdDB: -45,
|
||||||
|
}, transcriber, func(event Event) error {
|
||||||
|
events = append(events, event)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
|
||||||
|
if err := processor.Accept(constantSamples(audio.SampleRate, 0.5)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
transcriber.speechActive = false
|
||||||
|
if err := processor.Accept(make([]float32, audio.SamplesPerPacket())); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(events) != 2 || events[0].Kind != Provisional || events[1].Kind != Hide {
|
||||||
|
t.Fatalf("events = %#v, want Provisional then Hide", events)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVADPreviewRecognitionWindowIsBounded(t *testing.T) {
|
||||||
|
transcriber := &fakeTranscriber{text: "draft", speechActive: true}
|
||||||
|
processor := New(config.Settings{
|
||||||
|
Mode: config.ModeVAD,
|
||||||
|
ChunkDuration: time.Second,
|
||||||
|
PreviewThresholdDB: -45,
|
||||||
|
}, transcriber, func(Event) error { return nil })
|
||||||
|
|
||||||
|
for range 15 {
|
||||||
|
if err := processor.Accept(constantSamples(audio.SampleRate, 0.5)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
maxSamples := audio.DurationSamples(maxPreviewDuration)
|
||||||
|
for _, size := range transcriber.decodeSizes {
|
||||||
|
if size > maxSamples {
|
||||||
|
t.Fatalf("preview decode used %d samples, limit is %d", size, maxSamples)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(transcriber.decodeSizes) != 15 {
|
||||||
|
t.Fatalf("preview decode count = %d, want 15", len(transcriber.decodeSizes))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestFlushFlushesVAD(t *testing.T) {
|
func TestFlushFlushesVAD(t *testing.T) {
|
||||||
transcriber := &fakeTranscriber{}
|
transcriber := &fakeTranscriber{}
|
||||||
processor := New(config.Settings{Mode: config.ModeVAD, ChunkDuration: time.Second}, transcriber, func(Event) error { return nil })
|
processor := New(config.Settings{Mode: config.ModeVAD, ChunkDuration: time.Second}, transcriber, func(Event) error { return nil })
|
||||||
@@ -134,3 +238,11 @@ func TestFlushFlushesVAD(t *testing.T) {
|
|||||||
t.Fatal("VAD was not flushed")
|
t.Fatal("VAD was not flushed")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func constantSamples(count int, value float32) []float32 {
|
||||||
|
samples := make([]float32, count)
|
||||||
|
for i := range samples {
|
||||||
|
samples[i] = value
|
||||||
|
}
|
||||||
|
return samples
|
||||||
|
}
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ import (
|
|||||||
|
|
||||||
const (
|
const (
|
||||||
DefaultModelsDir = "models"
|
DefaultModelsDir = "models"
|
||||||
|
DefaultVADThreshold = 0.5
|
||||||
DefaultOverlayFontFamily = "Sans"
|
DefaultOverlayFontFamily = "Sans"
|
||||||
MinimumOverlayFinalLifetime = 3 * time.Second
|
MinimumOverlayFinalLifetime = 3 * time.Second
|
||||||
DefaultModelTimeout = 30 * time.Second
|
DefaultModelTimeout = 30 * time.Second
|
||||||
@@ -29,6 +30,7 @@ type Settings struct {
|
|||||||
ModelsDir string
|
ModelsDir string
|
||||||
Threads int
|
Threads int
|
||||||
PreviewThresholdDB float64
|
PreviewThresholdDB float64
|
||||||
|
VADThreshold float64
|
||||||
ModelAutoRestart bool
|
ModelAutoRestart bool
|
||||||
ModelTimeout time.Duration
|
ModelTimeout time.Duration
|
||||||
ModelShutdownTimeout time.Duration
|
ModelShutdownTimeout time.Duration
|
||||||
@@ -41,7 +43,8 @@ func Parse() Settings {
|
|||||||
flag.DurationVar(&settings.ChunkDuration, "chunk-duration", time.Second, "fixed chunk size or VAD preview refresh interval")
|
flag.DurationVar(&settings.ChunkDuration, "chunk-duration", time.Second, "fixed chunk size or VAD preview refresh interval")
|
||||||
flag.StringVar(&settings.ModelsDir, "models-dir", DefaultModelsDir, "directory containing downloaded Sherpa models")
|
flag.StringVar(&settings.ModelsDir, "models-dir", DefaultModelsDir, "directory containing downloaded Sherpa models")
|
||||||
flag.IntVar(&settings.Threads, "threads", 2, "CPU threads used by recognition and VAD")
|
flag.IntVar(&settings.Threads, "threads", 2, "CPU threads used by recognition and VAD")
|
||||||
flag.Float64Var(&settings.PreviewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS threshold used to begin VAD previews")
|
flag.Float64Var(&settings.PreviewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS gate below which recognition stays idle")
|
||||||
|
flag.Float64Var(&settings.VADThreshold, "vad-threshold", DefaultVADThreshold, "speech detection sensitivity from 0 to 1; higher rejects more non-speech")
|
||||||
flag.BoolVar(&settings.ModelAutoRestart, "model-auto-restart", true, "restart the isolated model worker after a crash or timeout")
|
flag.BoolVar(&settings.ModelAutoRestart, "model-auto-restart", true, "restart the isolated model worker after a crash or timeout")
|
||||||
flag.DurationVar(&settings.ModelTimeout, "model-timeout", DefaultModelTimeout, "maximum model startup or processing time; zero disables hang detection")
|
flag.DurationVar(&settings.ModelTimeout, "model-timeout", DefaultModelTimeout, "maximum model startup or processing time; zero disables hang detection")
|
||||||
flag.DurationVar(&settings.ModelShutdownTimeout, "model-shutdown-timeout", DefaultModelShutdownTimeout, "time allowed for model flush and cleanup before force-killing the worker")
|
flag.DurationVar(&settings.ModelShutdownTimeout, "model-shutdown-timeout", DefaultModelShutdownTimeout, "time allowed for model flush and cleanup before force-killing the worker")
|
||||||
@@ -94,7 +97,8 @@ func ParseDesktop() DesktopSettings {
|
|||||||
flags.DurationVar(&desktop.ChunkDuration, "chunk-duration", time.Second, "fixed chunk size or VAD preview refresh interval")
|
flags.DurationVar(&desktop.ChunkDuration, "chunk-duration", time.Second, "fixed chunk size or VAD preview refresh interval")
|
||||||
flags.StringVar(&desktop.ModelsDir, "models-dir", DefaultModelsDir, "directory containing downloaded Sherpa models")
|
flags.StringVar(&desktop.ModelsDir, "models-dir", DefaultModelsDir, "directory containing downloaded Sherpa models")
|
||||||
flags.IntVar(&desktop.Threads, "threads", 2, "CPU threads used by recognition and VAD")
|
flags.IntVar(&desktop.Threads, "threads", 2, "CPU threads used by recognition and VAD")
|
||||||
flags.Float64Var(&desktop.PreviewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS threshold used to begin VAD previews")
|
flags.Float64Var(&desktop.PreviewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS gate below which recognition stays idle")
|
||||||
|
flags.Float64Var(&desktop.VADThreshold, "vad-threshold", DefaultVADThreshold, "speech detection sensitivity from 0 to 1; higher rejects more non-speech")
|
||||||
flags.BoolVar(&desktop.ModelAutoRestart, "model-auto-restart", true, "restart the isolated model worker after a crash or timeout")
|
flags.BoolVar(&desktop.ModelAutoRestart, "model-auto-restart", true, "restart the isolated model worker after a crash or timeout")
|
||||||
flags.DurationVar(&desktop.ModelTimeout, "model-timeout", DefaultModelTimeout, "maximum model startup or processing time; zero disables hang detection")
|
flags.DurationVar(&desktop.ModelTimeout, "model-timeout", DefaultModelTimeout, "maximum model startup or processing time; zero disables hang detection")
|
||||||
flags.DurationVar(&desktop.ModelShutdownTimeout, "model-shutdown-timeout", DefaultModelShutdownTimeout, "time allowed for model flush and cleanup before force-killing the worker")
|
flags.DurationVar(&desktop.ModelShutdownTimeout, "model-shutdown-timeout", DefaultModelShutdownTimeout, "time allowed for model flush and cleanup before force-killing the worker")
|
||||||
@@ -125,6 +129,9 @@ func (s Settings) Validate() error {
|
|||||||
if s.Threads <= 0 {
|
if s.Threads <= 0 {
|
||||||
return errors.New("--threads must be positive")
|
return errors.New("--threads must be positive")
|
||||||
}
|
}
|
||||||
|
if s.VADThreshold < 0 || s.VADThreshold > 1 {
|
||||||
|
return errors.New("--vad-threshold must be between 0 and 1")
|
||||||
|
}
|
||||||
if s.ModelTimeout < 0 {
|
if s.ModelTimeout < 0 {
|
||||||
return errors.New("--model-timeout cannot be negative")
|
return errors.New("--model-timeout cannot be negative")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -59,6 +59,8 @@ func TestDesktopSettingsValidate(t *testing.T) {
|
|||||||
{"short timeout", func(s *DesktopSettings) { s.Overlay.FinalTimeout = time.Second }},
|
{"short timeout", func(s *DesktopSettings) { s.Overlay.FinalTimeout = time.Second }},
|
||||||
{"negative model timeout", func(s *DesktopSettings) { s.ModelTimeout = -time.Second }},
|
{"negative model timeout", func(s *DesktopSettings) { s.ModelTimeout = -time.Second }},
|
||||||
{"zero model shutdown timeout", func(s *DesktopSettings) { s.ModelShutdownTimeout = 0 }},
|
{"zero model shutdown timeout", func(s *DesktopSettings) { s.ModelShutdownTimeout = 0 }},
|
||||||
|
{"negative VAD threshold", func(s *DesktopSettings) { s.VADThreshold = -0.1 }},
|
||||||
|
{"large VAD threshold", func(s *DesktopSettings) { s.VADThreshold = 1.1 }},
|
||||||
}
|
}
|
||||||
for _, test := range tests {
|
for _, test := range tests {
|
||||||
t.Run(test.name, func(t *testing.T) {
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
|||||||
@@ -57,7 +57,7 @@ func New(settings config.Settings) (*Resources, error) {
|
|||||||
|
|
||||||
resources := &Resources{recognizer: recognizer}
|
resources := &Resources{recognizer: recognizer}
|
||||||
if settings.Mode == config.ModeVAD {
|
if settings.Mode == config.ModeVAD {
|
||||||
resources.vad = newVAD(paths.vad, settings.Threads)
|
resources.vad = newVAD(paths.vad, settings.Threads, settings.VADThreshold)
|
||||||
if resources.vad == nil {
|
if resources.vad == nil {
|
||||||
resources.Close()
|
resources.Close()
|
||||||
return nil, fmt.Errorf("create Silero VAD: Sherpa returned nil")
|
return nil, fmt.Errorf("create Silero VAD: Sherpa returned nil")
|
||||||
@@ -92,6 +92,10 @@ func (r *Resources) AcceptVAD(samples []float32) {
|
|||||||
r.vad.AcceptWaveform(samples)
|
r.vad.AcceptWaveform(samples)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (r *Resources) SpeechActive() bool {
|
||||||
|
return r.vad.IsSpeech()
|
||||||
|
}
|
||||||
|
|
||||||
func (r *Resources) FlushVAD() {
|
func (r *Resources) FlushVAD() {
|
||||||
r.vad.Flush()
|
r.vad.Flush()
|
||||||
}
|
}
|
||||||
@@ -132,11 +136,11 @@ func validatePaths(modelsDir string, mode config.Mode) (modelPaths, error) {
|
|||||||
return paths, nil
|
return paths, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func newVAD(model string, threads int) *sherpa.VoiceActivityDetector {
|
func newVAD(model string, threads int, threshold float64) *sherpa.VoiceActivityDetector {
|
||||||
return sherpa.NewVoiceActivityDetector(&sherpa.VadModelConfig{
|
return sherpa.NewVoiceActivityDetector(&sherpa.VadModelConfig{
|
||||||
SileroVad: sherpa.SileroVadModelConfig{
|
SileroVad: sherpa.SileroVadModelConfig{
|
||||||
Model: model,
|
Model: model,
|
||||||
Threshold: 0.5,
|
Threshold: float32(threshold),
|
||||||
// A 500 ms pause ends the current caption line and starts the next one.
|
// A 500 ms pause ends the current caption line and starts the next one.
|
||||||
MinSilenceDuration: 0.5,
|
MinSilenceDuration: 0.5,
|
||||||
MinSpeechDuration: 0.25,
|
MinSpeechDuration: 0.25,
|
||||||
|
|||||||
Reference in New Issue
Block a user