2 Commits
Author SHA1 Message Date
kato 55fbd357b0 docs: update README with VAD, new threshold option, and silence handling
Release / Tests before release (push) Successful in 1m29s
Release / Build and publish release (push) Successful in 3m5s
Tests / Go and GTK tests (push) Successful in 1m39s
Update README to clarify VAD mode behavior: provisional caption only after Silero detects speech, Parakeet idle during silence. Document new --vad-threshold option to control Silero speech confidence. Add troubleshooting tip for music/noise false detections.
2026-07-17 10:35:59 +03:00
kato b0487be39e feat: implement isolated model worker with auto-restart and hot-reload
Tests / Go and GTK tests (push) Successful in 1m38s
Release / Tests before release (push) Successful in 1m34s
Release / Build and publish release (push) Successful in 2m24s
Introduce a supervised child process for capture, VAD, and recognition,
enabling automatic restart on crash/hang and hot-reload via SIGHUP.
Add `--model-auto-restart`, `--model-timeout`, and
`--model-shutdown-timeout` options. Refactor `Run()` with lifecycle
hooks for proper process group cleanup. Update README with new
behavior and CLI flags.
2026-07-17 01:47:37 +03:00
14 changed files with 907 additions and 56 deletions
+23 -3
View File
@@ -23,7 +23,7 @@ This creates `models/parakeet-tdt-v2/` and `models/silero_vad_v5.onnx`. They are
## Terminal captions ## Terminal captions
VAD mode is recommended for normal use. It redraws a provisional caption while speech is active and commits a timestamped caption after the utterance ends: VAD mode is recommended for normal use. It redraws a provisional caption only after Silero detects speech and commits a timestamped caption after the utterance ends:
```bash ```bash
go run ./src/cmd/captioneer --mode=vad go run ./src/cmd/captioneer --mode=vad
@@ -50,6 +50,14 @@ Final captions use stdout; diagnostics and overload warnings use stderr. This ma
Press <kbd>Ctrl</kbd>+<kbd>C</kbd> to stop. Captioneer stops `parec` and flushes pending audio before exiting. Press <kbd>Ctrl</kbd>+<kbd>C</kbd> to stop. Captioneer stops `parec` and flushes pending audio before exiting.
Sherpa recognition, Silero VAD, and audio capture run in an isolated worker process. The terminal or GTK process stays responsive if native inference crashes or hangs: Captioneer restarts the worker automatically after a failure, and Ctrl+C force-kills only that worker if it cannot flush within the configured shutdown timeout. Send `SIGHUP` to the parent Captioneer PID to hot-reload the models without closing the terminal or overlay:
```bash
kill -HUP <captioneer-parent-pid>
```
An automatic or manual restart begins a fresh capture/transcription session; audio being processed by the failed worker cannot be recovered.
## Desktop overlay ## Desktop overlay
The simplest Bazzite development command is: The simplest Bazzite development command is:
@@ -66,6 +74,8 @@ Use `--output=both` to keep terminal output as well:
In VAD mode, a 500 ms pause finalizes the current speech segment. Captioneer splits that transcript again at sentence punctuation and at word-safe boundaries of at most 64 characters, so continuous speech does not become one enormous paragraph. The live provisional view keeps the latest three digestible chunks and grows vertically without ellipsizing them. Completed chunks remain visible for at least three seconds, receive extra time based on their word count, and gradually fade from white to a restrained gray as they age. When several chunks finalize together, their removal is staggered in reading order instead of deleting the whole block at once. Up to six completed chunks remain above the provisional text; if a seventh arrives, the oldest is removed. Blank recognition results never erase completed text before its reading lifetime expires. In VAD mode, a 500 ms pause finalizes the current speech segment. Captioneer splits that transcript again at sentence punctuation and at word-safe boundaries of at most 64 characters, so continuous speech does not become one enormous paragraph. The live provisional view keeps the latest three digestible chunks and grows vertically without ellipsizing them. Completed chunks remain visible for at least three seconds, receive extra time based on their word count, and gradually fade from white to a restrained gray as they age. When several chunks finalize together, their removal is staggered in reading order instead of deleting the whole block at once. Up to six completed chunks remain above the provisional text; if a seventh arrives, the oldest is removed. Blank recognition results never erase completed text before its reading lifetime expires.
The expensive Parakeet recognizer stays idle during silence and audio that Silero does not classify as speech. A small rolling preview window also prevents long continuous audio from making each refresh progressively slower. Silero itself continues receiving the live stream because it needs the quiet frames to detect when speech ends and when it starts again.
Text is left-aligned inside the centered, fixed-width dark bubble and wraps at its configured width instead of extending off-screen. Pointer input passes through the window by default. Text is left-aligned inside the centered, fixed-width dark bubble and wraps at its configured width instead of extending off-screen. Pointer input passes through the window by default.
To build outside the helper script, install GTK4 and GTK4 Layer Shell development packages, then run: To build outside the helper script, install GTK4 and GTK4 Layer Shell development packages, then run:
@@ -97,7 +107,11 @@ Layer Shell and X11 hints are requests, not universal guarantees. Same-layer win
| `--chunk-duration=1s` | `1s` | Fixed chunk length or VAD preview refresh interval. Accepts Go durations such as `500ms` or `2s`. | | `--chunk-duration=1s` | `1s` | Fixed chunk length or VAD preview refresh interval. Accepts Go durations such as `500ms` or `2s`. |
| `--models-dir=models` | `models` | Directory containing `parakeet-tdt-v2/` and `silero_vad_v5.onnx`. | | `--models-dir=models` | `models` | Directory containing `parakeet-tdt-v2/` and `silero_vad_v5.onnx`. |
| `--threads=2` | `2` | CPU threads used by recognition and VAD. | | `--threads=2` | `2` | CPU threads used by recognition and VAD. |
| `--preview-threshold-dbfs=-45` | `-45` | RMS threshold that starts a provisional VAD caption. Raise it to reject background audio; lower it for quiet speech. | | `--preview-threshold-dbfs=-45` | `-45` | RMS gate below which recognition stays idle. Raise it to reject quiet background noise; lower it for quiet speech. In fixed mode, gated chunks are not sent to Parakeet. |
| `--vad-threshold=0.5` | `0.5` | Silero speech confidence from `0` to `1`. Raise it to reject more music/noise; lower it if real speech is missed. Applies to VAD mode. |
| `--model-auto-restart=true` | `true` | Restart the isolated model worker after a crash, capture failure, or timeout. Set `false` to report the failure and exit instead. |
| `--model-timeout=30s` | `30s` | Maximum model startup, continuously busy processing time, or missing-heartbeat period before the worker is considered hung. `0s` disables hang detection. |
| `--model-shutdown-timeout=2s` | `2s` | Time allowed for pending-audio flush and model cleanup after Ctrl+C or reload before the worker is force-killed. |
### Desktop-only options ### Desktop-only options
@@ -127,6 +141,10 @@ Examples:
# Fixed two-second captions on a named monitor # Fixed two-second captions on a named monitor
./scripts/distrobox/run.sh --mode=fixed --chunk-duration=2s \ ./scripts/distrobox/run.sh --mode=fixed --chunk-duration=2s \
--output=overlay --overlay-monitor=DP-1 --output=overlay --overlay-monitor=DP-1
# Be more conservative when instrumental music causes false speech detections
./scripts/distrobox/run.sh --mode=vad --output=overlay \
--vad-threshold=0.65 --preview-threshold-dbfs=-40
``` ```
## Bazzite development with Distrobox ## Bazzite development with Distrobox
@@ -162,14 +180,16 @@ Captioneer itself has no environment-based application configuration; use the co
- **Missing model files:** run `./scripts/download-models.sh` or correct `--models-dir`. - **Missing model files:** run `./scripts/download-models.sh` or correct `--models-dir`.
- **`pactl` or `parec` missing:** install PulseAudio utilities, or use the Distrobox scripts. - **`pactl` or `parec` missing:** install PulseAudio utilities, or use the Distrobox scripts.
- **No captions:** confirm the current sink with `pactl get-default-sink`; Captioneer listens to `<sink>.monitor`. - **No captions:** confirm the current sink with `pactl get-default-sink`; Captioneer listens to `<sink>.monitor`.
- **Music/noise produces captions:** use VAD mode and try `--vad-threshold=0.65`. Increase it in small steps if false detections continue. `--preview-threshold-dbfs=-40` can also reject quiet background audio, but an RMS gate cannot distinguish loud music from speech. Singing may still be treated as speech.
- **Processing overload warning:** inference fell behind and the oldest 100 ms packet was dropped. Try more `--threads` or a longer `--chunk-duration`. - **Processing overload warning:** inference fell behind and the oldest 100 ms packet was dropped. Try more `--threads` or a longer `--chunk-duration`.
- **Overlay is not always above other windows:** read the backend warning. On Wayland this usually means the compositor lacks Layer Shell; add a compositor rule for the `Captioneer Overlay` window if supported. - **Overlay is not always above other windows:** read the backend warning. On Wayland this usually means the compositor lacks Layer Shell; add a compositor rule for the `Captioneer Overlay` window if supported.
- **Overlay initialization fails:** `--output=overlay` exits with the error. `--output=both` warns and continues terminal-only. - **Overlay initialization fails:** `--output=overlay` exits with the error. `--output=both` warns and continues terminal-only.
- **A locally built desktop binary misses shared libraries:** use `scripts/distrobox/run.sh` or install the matching GTK4 and GTK4 Layer Shell runtime packages. The desktop command in a published release carries these application libraries in its private `lib/` directory; keep the extracted directory together. - **A locally built desktop binary misses shared libraries:** use `scripts/distrobox/run.sh` or install the matching GTK4 and GTK4 Layer Shell runtime packages. The desktop command in a published release carries these application libraries in its private `lib/` directory; keep the extracted directory together.
- **The model worker repeatedly restarts:** inspect the warning on stderr. Slow machines may need a larger `--model-timeout`; use `--model-auto-restart=false` when diagnosing a persistent crash so Captioneer exits on the first failure.
## Development ## Development
The transcription pipeline emits presentation-neutral events. `output/terminal` and `output/overlay` consume those events independently; GTK stays on its main OS thread while capture and recognition run on a worker. The transcription pipeline emits presentation-neutral events. `output/terminal` and `output/overlay` consume those events independently; GTK stays on its main OS thread while capture, VAD, and recognition run in a supervised child process. Caption events and health messages cross a private pipe, and the child is placed in its own process group so its `parec` process can be stopped or replaced with it.
```text ```text
src/app/ capture/model/process lifecycle src/app/ capture/model/process lifecycle
+36 -6
View File
@@ -14,15 +14,28 @@ import (
"tea.chunkbyte.com/kato/captioneer/src/output" "tea.chunkbyte.com/kato/captioneer/src/output"
) )
func Run(ctx context.Context, settings config.Settings, sink output.Sink, diagnostics io.Writer) error { type directHooks struct {
if err := capture.ValidatePrograms(); err != nil { ready func() error
busy func() error
idle func() error
}
func runDirect(ctx context.Context, settings config.Settings, sink output.Sink, diagnostics io.Writer, hooks directHooks) error {
if err := hooks.busy(); err != nil {
return err return err
} }
resources, err := models.New(settings) resources, err := models.New(settings)
if idleErr := hooks.idle(); err == nil {
err = idleErr
}
if err != nil { if err != nil {
return err return err
} }
defer resources.Close() defer func() {
_ = hooks.busy()
resources.Close()
_ = hooks.idle()
}()
ctx, cancel := context.WithCancel(ctx) ctx, cancel := context.WithCancel(ctx)
defer cancel() defer cancel()
@@ -32,24 +45,41 @@ func Run(ctx context.Context, settings config.Settings, sink output.Sink, diagno
return fmt.Errorf("find default output monitor: %w", err) return fmt.Errorf("find default output monitor: %w", err)
} }
fmt.Fprintf(diagnostics, "Capturing from: %s\n", monitorSource) fmt.Fprintf(diagnostics, "Capturing from: %s\n", monitorSource)
fmt.Fprintln(diagnostics, "Press Ctrl+C to stop.")
packets, waitCapture, err := capture.Packets(ctx, monitorSource) packets, waitCapture, err := capture.Packets(ctx, monitorSource)
if err != nil { if err != nil {
return fmt.Errorf("start system-audio capture: %w", err) return fmt.Errorf("start system-audio capture: %w", err)
} }
if err := hooks.ready(); err != nil {
cancel()
_ = waitCapture()
return err
}
processor := captions.New(settings, resources, sink.Publish) processor := captions.New(settings, resources, sink.Publish)
var processErr error var processErr error
for packet := range packets { for packet := range packets {
if err := processor.Accept(packet); err != nil { if err := hooks.busy(); err != nil {
processErr = err
cancel()
break
}
acceptErr := processor.Accept(packet)
idleErr := hooks.idle()
if acceptErr != nil || idleErr != nil {
err := errors.Join(acceptErr, idleErr)
processErr = fmt.Errorf("publish caption: %w", err) processErr = fmt.Errorf("publish caption: %w", err)
cancel() cancel()
break break
} }
} }
if processErr == nil { if processErr == nil {
processErr = processor.Flush() if err := hooks.busy(); err != nil {
processErr = err
} else {
processErr = processor.Flush()
processErr = errors.Join(processErr, hooks.idle())
}
} }
captureErr := waitCapture() captureErr := waitCapture()
if captureErr != nil && ctx.Err() == nil { if captureErr != nil && ctx.Err() == nil {
+361
View File
@@ -0,0 +1,361 @@
package app
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/exec"
"os/signal"
"strings"
"syscall"
"time"
"tea.chunkbyte.com/kato/captioneer/src/capture"
"tea.chunkbyte.com/kato/captioneer/src/config"
"tea.chunkbyte.com/kato/captioneer/src/models"
"tea.chunkbyte.com/kato/captioneer/src/output"
)
const modelRestartDelay = time.Second
type attemptResult uint8
const (
attemptStopped attemptResult = iota + 1
attemptReload
attemptFailed
attemptFatal
)
type workerHealth struct {
startedAt time.Time
lastSeen time.Time
ready bool
busySince time.Time
}
func (h *workerHealth) observe(messageType string, now time.Time) {
h.lastSeen = now
switch messageType {
case messageReady:
h.ready = true
h.busySince = time.Time{}
case messageBusy:
if h.busySince.IsZero() {
h.busySince = now
}
case messageIdle:
h.busySince = time.Time{}
}
}
func (h workerHealth) timeoutError(now time.Time, timeout time.Duration) error {
if timeout == 0 {
return nil
}
if !h.ready && now.Sub(h.startedAt) >= timeout {
return fmt.Errorf("model worker did not become ready within %s", timeout)
}
if !h.busySince.IsZero() && now.Sub(h.busySince) >= timeout {
return fmt.Errorf("model operation exceeded %s", timeout)
}
if h.ready && now.Sub(h.lastSeen) >= timeout {
return fmt.Errorf("model worker heartbeat was silent for %s", timeout)
}
return nil
}
// Run supervises an isolated process containing capture, VAD, and recognition.
// Presentation stays in this process, so the native model can be replaced
// without restarting the terminal or GTK main loop.
func Run(ctx context.Context, settings config.Settings, sink output.Sink, diagnostics io.Writer) error {
if err := capture.ValidatePrograms(); err != nil {
return err
}
if err := models.Validate(settings); err != nil {
return err
}
fmt.Fprintln(diagnostics, "Press Ctrl+C to stop. Send SIGHUP to hot-reload the model worker.")
reload := make(chan os.Signal, 1)
signal.Notify(reload, syscall.SIGHUP)
defer signal.Stop(reload)
for {
result, err := runWorkerAttempt(ctx, settings, sink, diagnostics, reload)
if ctx.Err() != nil {
return nil
}
switch result {
case attemptStopped:
return nil
case attemptFatal:
return err
case attemptReload:
fmt.Fprintln(diagnostics, "Reloading model worker.")
continue
case attemptFailed:
if !settings.ModelAutoRestart {
return err
}
fmt.Fprintf(diagnostics, "warning: %v; restarting model worker in %s\n", err, modelRestartDelay)
}
timer := time.NewTimer(modelRestartDelay)
select {
case <-ctx.Done():
if !timer.Stop() {
<-timer.C
}
return nil
case <-reload:
if !timer.Stop() {
<-timer.C
}
fmt.Fprintln(diagnostics, "Reloading model worker.")
case <-timer.C:
}
}
}
func runWorkerAttempt(
ctx context.Context,
settings config.Settings,
sink output.Sink,
diagnostics io.Writer,
reload <-chan os.Signal,
) (attemptResult, error) {
cmd, protocol, err := startModelWorker(settings, diagnostics)
if err != nil {
return attemptFailed, fmt.Errorf("start model worker: %w", err)
}
defer protocol.Close()
messages := make(chan workerMessage)
protocolDone := make(chan error, 1)
go readWorkerMessages(protocol, messages, protocolDone)
wait := make(chan error, 1)
go func() { wait <- cmd.Wait() }()
checkInterval := 250 * time.Millisecond
if settings.ModelTimeout > 0 && settings.ModelTimeout < checkInterval {
checkInterval = settings.ModelTimeout / 2
if checkInterval <= 0 {
checkInterval = time.Millisecond
}
}
ticker := time.NewTicker(checkInterval)
defer ticker.Stop()
startedAt := time.Now()
health := workerHealth{startedAt: startedAt, lastSeen: startedAt}
messageChannel := (<-chan workerMessage)(messages)
waitChannel := (<-chan error)(wait)
contextDone := ctx.Done()
var stopTimer *time.Timer
var stopDeadline <-chan time.Time
var stopping bool
var desiredResult attemptResult
var desiredErr error
var processDone bool
var processErr error
var protocolClosed bool
var protocolErr error
var workerFailure string
beginStop := func(result attemptResult, stopErr error) {
if stopping {
if result == attemptFatal {
desiredResult = result
desiredErr = errors.Join(desiredErr, stopErr)
}
return
}
stopping = true
desiredResult = result
desiredErr = stopErr
if err := signalWorkerGroup(cmd, syscall.SIGTERM); err != nil {
desiredErr = errors.Join(desiredErr, err)
}
stopTimer = time.NewTimer(settings.ModelShutdownTimeout)
stopDeadline = stopTimer.C
}
for {
if processDone && protocolClosed {
if stopTimer != nil && !stopTimer.Stop() {
select {
case <-stopTimer.C:
default:
}
}
if stopping {
return desiredResult, desiredErr
}
// The worker may have crashed before its parec child. A process group
// remains signalable after its leader exits, so clean up any survivors
// before starting a replacement worker.
cleanupErr := signalWorkerGroup(cmd, syscall.SIGKILL)
if protocolErr != nil {
return attemptFailed, errors.Join(fmt.Errorf("model worker protocol: %w", protocolErr), cleanupErr)
}
if workerFailure != "" {
return attemptFailed, errors.Join(errors.New(workerFailure), cleanupErr)
}
if processErr != nil {
return attemptFailed, errors.Join(fmt.Errorf("model worker exited: %w", processErr), cleanupErr)
}
return attemptFailed, errors.Join(errors.New("model worker exited unexpectedly"), cleanupErr)
}
select {
case <-contextDone:
contextDone = nil
beginStop(attemptStopped, nil)
case <-reload:
beginStop(attemptReload, nil)
case message, ok := <-messageChannel:
if !ok {
messageChannel = nil
protocolClosed = true
protocolErr = <-protocolDone
if !processDone && !stopping {
beginStop(attemptFailed, errors.New("model worker protocol closed unexpectedly"))
}
continue
}
now := time.Now()
health.observe(message.Type, now)
switch message.Type {
case messageReady:
fmt.Fprintln(diagnostics, "Model worker ready.")
case messageBusy, messageIdle:
case messageHeartbeat:
case messageCaption:
if message.Event == nil {
beginStop(attemptFailed, errors.New("model worker sent an empty caption event"))
continue
}
if err := sink.Publish(*message.Event); err != nil {
beginStop(attemptFatal, fmt.Errorf("publish caption: %w", err))
}
case messageFailure:
workerFailure = message.Error
default:
beginStop(attemptFailed, fmt.Errorf("unknown model worker message %q", message.Type))
}
case err := <-waitChannel:
waitChannel = nil
processDone = true
processErr = err
case now := <-ticker.C:
if !stopping {
if err := health.timeoutError(now, settings.ModelTimeout); err != nil {
beginStop(attemptFailed, err)
}
}
case <-stopDeadline:
stopDeadline = nil
if !processDone {
fmt.Fprintf(diagnostics, "warning: model worker did not stop within %s; force-killing it\n", settings.ModelShutdownTimeout)
if err := signalWorkerGroup(cmd, syscall.SIGKILL); err != nil {
desiredErr = errors.Join(desiredErr, err)
}
}
}
}
}
func startModelWorker(settings config.Settings, diagnostics io.Writer) (*exec.Cmd, *os.File, error) {
executable, err := os.Executable()
if err != nil {
return nil, nil, err
}
encodedSettings, err := encodeWorkerSettings(settings)
if err != nil {
return nil, nil, err
}
protocolRead, protocolWrite, err := os.Pipe()
if err != nil {
return nil, nil, err
}
cmd := exec.Command(executable)
cmd.Env = workerEnvironment(encodedSettings)
cmd.ExtraFiles = []*os.File{protocolWrite}
cmd.Stdout = diagnostics
cmd.Stderr = diagnostics
cmd.SysProcAttr = &syscall.SysProcAttr{
Setpgid: true,
Pdeathsig: syscall.SIGKILL,
}
if err := cmd.Start(); err != nil {
protocolRead.Close()
protocolWrite.Close()
return nil, nil, err
}
if err := protocolWrite.Close(); err != nil {
_ = signalWorkerGroup(cmd, syscall.SIGKILL)
_ = cmd.Wait()
protocolRead.Close()
return nil, nil, err
}
return cmd, protocolRead, nil
}
func workerEnvironment(encodedSettings string) []string {
environment := make([]string, 0, len(os.Environ())+2)
for _, item := range os.Environ() {
if strings.HasPrefix(item, modelWorkerEnvironment+"=") ||
strings.HasPrefix(item, modelSettingsEnvironment+"=") {
continue
}
environment = append(environment, item)
}
return append(environment,
modelWorkerEnvironment+"=1",
modelSettingsEnvironment+"="+encodedSettings,
)
}
func readWorkerMessages(protocol io.Reader, messages chan<- workerMessage, done chan<- error) {
defer close(messages)
decoder := json.NewDecoder(protocol)
for {
var message workerMessage
if err := decoder.Decode(&message); err != nil {
if errors.Is(err, io.EOF) {
done <- nil
} else {
done <- err
}
return
}
messages <- message
}
}
func signalWorkerGroup(cmd *exec.Cmd, signal syscall.Signal) error {
if cmd.Process == nil {
return nil
}
err := syscall.Kill(-cmd.Process.Pid, signal)
if errors.Is(err, syscall.ESRCH) {
return nil
}
if err != nil {
return fmt.Errorf("signal model worker: %w", err)
}
return nil
}
+67
View File
@@ -0,0 +1,67 @@
package app
import (
"strings"
"testing"
"time"
"tea.chunkbyte.com/kato/captioneer/src/config"
)
func TestWorkerHealthDetectsStartupAndOperationTimeouts(t *testing.T) {
started := time.Unix(100, 0)
health := workerHealth{startedAt: started, lastSeen: started}
if err := health.timeoutError(started.Add(time.Second), 2*time.Second); err != nil {
t.Fatalf("healthy startup reported a timeout: %v", err)
}
if err := health.timeoutError(started.Add(2*time.Second), 2*time.Second); err == nil ||
!strings.Contains(err.Error(), "did not become ready") {
t.Fatalf("startup timeout = %v", err)
}
health.observe(messageReady, started.Add(2*time.Second))
health.observe(messageBusy, started.Add(3*time.Second))
if err := health.timeoutError(started.Add(4*time.Second), 2*time.Second); err != nil {
t.Fatalf("healthy operation reported a timeout: %v", err)
}
if err := health.timeoutError(started.Add(5*time.Second), 2*time.Second); err == nil ||
!strings.Contains(err.Error(), "operation exceeded") {
t.Fatalf("operation timeout = %v", err)
}
health.observe(messageIdle, started.Add(5*time.Second))
health.observe(messageHeartbeat, started.Add(6*time.Second))
if err := health.timeoutError(started.Add(7*time.Second), 2*time.Second); err != nil {
t.Fatalf("idle worker reported a timeout: %v", err)
}
if err := health.timeoutError(started.Add(8*time.Second), 2*time.Second); err == nil ||
!strings.Contains(err.Error(), "heartbeat was silent") {
t.Fatalf("heartbeat timeout = %v", err)
}
}
func TestWorkerSettingsRoundTrip(t *testing.T) {
want := config.Settings{
Mode: config.ModeVAD,
ChunkDuration: time.Second,
ModelsDir: "models with spaces",
Threads: 3,
PreviewThresholdDB: -42,
VADThreshold: 0.65,
ModelAutoRestart: true,
ModelTimeout: 12 * time.Second,
ModelShutdownTimeout: 2 * time.Second,
}
encoded, err := encodeWorkerSettings(want)
if err != nil {
t.Fatal(err)
}
got, err := decodeWorkerSettings(encoded)
if err != nil {
t.Fatal(err)
}
if got != want {
t.Fatalf("settings round trip = %#v, want %#v", got, want)
}
}
+152
View File
@@ -0,0 +1,152 @@
package app
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/signal"
"sync"
"syscall"
"time"
"tea.chunkbyte.com/kato/captioneer/src/captions"
"tea.chunkbyte.com/kato/captioneer/src/config"
)
const (
modelWorkerEnvironment = "CAPTIONEER_INTERNAL_MODEL_WORKER"
modelSettingsEnvironment = "CAPTIONEER_INTERNAL_MODEL_SETTINGS"
modelProtocolFD = 3
)
const (
messageReady = "ready"
messageBusy = "busy"
messageIdle = "idle"
messageCaption = "caption"
messageFailure = "failure"
messageHeartbeat = "heartbeat"
)
type workerMessage struct {
Type string `json:"type"`
Event *captions.Event `json:"event,omitempty"`
Error string `json:"error,omitempty"`
}
type workerReporter struct {
mu sync.Mutex
encoder *json.Encoder
}
func (r *workerReporter) send(message workerMessage) error {
r.mu.Lock()
defer r.mu.Unlock()
return r.encoder.Encode(message)
}
type protocolSink struct {
reporter *workerReporter
}
func (s protocolSink) Publish(event captions.Event) error {
return s.reporter.send(workerMessage{Type: messageCaption, Event: &event})
}
func (protocolSink) Close() error { return nil }
// IsModelWorker reports whether this process was created as Captioneer's
// private inference worker. The environment marker is intentionally internal.
func IsModelWorker() bool {
return os.Getenv(modelWorkerEnvironment) == "1"
}
func encodeWorkerSettings(settings config.Settings) (string, error) {
data, err := json.Marshal(settings)
if err != nil {
return "", err
}
return base64.RawStdEncoding.EncodeToString(data), nil
}
func decodeWorkerSettings(value string) (config.Settings, error) {
data, err := base64.RawStdEncoding.DecodeString(value)
if err != nil {
return config.Settings{}, fmt.Errorf("decode model worker settings: %w", err)
}
var settings config.Settings
if err := json.Unmarshal(data, &settings); err != nil {
return config.Settings{}, fmt.Errorf("parse model worker settings: %w", err)
}
if err := settings.Validate(); err != nil {
return config.Settings{}, fmt.Errorf("validate model worker settings: %w", err)
}
return settings, nil
}
// RunModelWorker owns all Sherpa/VAD resources in the supervised child. It
// returns a process exit code so command entrypoints can call os.Exit.
func RunModelWorker(diagnostics io.Writer) int {
protocol := os.NewFile(uintptr(modelProtocolFD), "captioneer-model-protocol")
if protocol == nil {
fmt.Fprintln(diagnostics, "captioneer: model worker protocol is unavailable")
return 1
}
defer protocol.Close()
syscall.CloseOnExec(modelProtocolFD)
settings, err := decodeWorkerSettings(os.Getenv(modelSettingsEnvironment))
if err != nil {
fmt.Fprintf(diagnostics, "captioneer: %v\n", err)
return 1
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
reporter := &workerReporter{encoder: json.NewEncoder(protocol)}
hooks := directHooks{
ready: func() error { return reporter.send(workerMessage{Type: messageReady}) },
busy: func() error { return reporter.send(workerMessage{Type: messageBusy}) },
idle: func() error { return reporter.send(workerMessage{Type: messageIdle}) },
}
heartbeatInterval := time.Second
if settings.ModelTimeout > 0 && settings.ModelTimeout < 4*heartbeatInterval {
heartbeatInterval = settings.ModelTimeout / 4
if heartbeatInterval <= 0 {
heartbeatInterval = time.Millisecond
}
}
heartbeatContext, stopHeartbeat := context.WithCancel(ctx)
heartbeatDone := make(chan struct{})
go func() {
defer close(heartbeatDone)
ticker := time.NewTicker(heartbeatInterval)
defer ticker.Stop()
for {
select {
case <-heartbeatContext.Done():
return
case <-ticker.C:
if reporter.send(workerMessage{Type: messageHeartbeat}) != nil {
return
}
}
}
}()
err = runDirect(ctx, settings, protocolSink{reporter: reporter}, diagnostics, hooks)
stopHeartbeat()
<-heartbeatDone
if err == nil || errors.Is(err, context.Canceled) {
return 0
}
_ = reporter.send(workerMessage{Type: messageFailure, Error: err.Error()})
fmt.Fprintf(diagnostics, "captioneer: model worker: %v\n", err)
return 1
}
+1
View File
@@ -20,6 +20,7 @@ type Event struct {
type Transcriber interface { type Transcriber interface {
Decode(samples []float32) string Decode(samples []float32) string
AcceptVAD(samples []float32) AcceptVAD(samples []float32)
SpeechActive() bool
FlushVAD() FlushVAD()
NextSpeechSegment() (start int, samples []float32, ok bool) NextSpeechSegment() (start int, samples []float32, ok bool)
} }
+66 -16
View File
@@ -3,22 +3,30 @@ package captions
import ( import (
"strings" "strings"
"time"
"tea.chunkbyte.com/kato/captioneer/src/audio" "tea.chunkbyte.com/kato/captioneer/src/audio"
"tea.chunkbyte.com/kato/captioneer/src/config" "tea.chunkbyte.com/kato/captioneer/src/config"
) )
// Provisional recognition uses only a recent window. Re-decoding an entire
// long utterance every refresh grows increasingly expensive, while the UI only
// presents the latest provisional lines. The final event still decodes the
// complete VAD segment.
const maxPreviewDuration = 10 * time.Second
type Processor struct { type Processor struct {
settings config.Settings settings config.Settings
transcriber Transcriber transcriber Transcriber
emit func(Event) error emit func(Event) error
fixedSamples []float32 fixedSamples []float32
previewSamples []float32 previewSamples []float32
previewActive bool previewActive bool
previewStart int previewStart int
nextPreviewAt int nextPreviewAt int
totalSamples int previewReceived int
totalSamples int
} }
func New(settings config.Settings, transcriber Transcriber, emit func(Event) error) *Processor { func New(settings config.Settings, transcriber Transcriber, emit func(Event) error) *Processor {
@@ -43,19 +51,26 @@ func (p *Processor) Accept(samples []float32) error {
func (p *Processor) Flush() error { func (p *Processor) Flush() error {
if p.settings.Mode == config.ModeFixed { if p.settings.Mode == config.ModeFixed {
if len(p.fixedSamples) > 0 { if len(p.fixedSamples) > 0 {
return p.emitFinal(p.totalSamples, p.totalSamples+len(p.fixedSamples), p.fixedSamples) return p.emitFixedFinal(p.totalSamples, p.totalSamples+len(p.fixedSamples), p.fixedSamples)
} }
return nil return nil
} }
p.transcriber.FlushVAD() p.transcriber.FlushVAD()
return p.drainVAD() if err := p.drainVAD(); err != nil {
return err
}
if p.previewActive {
p.resetPreview()
return p.emit(Event{Kind: Hide})
}
return nil
} }
func (p *Processor) acceptFixed(samples []float32) error { func (p *Processor) acceptFixed(samples []float32) error {
p.fixedSamples = append(p.fixedSamples, samples...) p.fixedSamples = append(p.fixedSamples, samples...)
chunkSize := audio.DurationSamples(p.settings.ChunkDuration) chunkSize := audio.DurationSamples(p.settings.ChunkDuration)
for len(p.fixedSamples) >= chunkSize { for len(p.fixedSamples) >= chunkSize {
if err := p.emitFinal(p.totalSamples, p.totalSamples+chunkSize, p.fixedSamples[:chunkSize]); err != nil { if err := p.emitFixedFinal(p.totalSamples, p.totalSamples+chunkSize, p.fixedSamples[:chunkSize]); err != nil {
return err return err
} }
p.fixedSamples = p.fixedSamples[chunkSize:] p.fixedSamples = p.fixedSamples[chunkSize:]
@@ -68,20 +83,33 @@ func (p *Processor) acceptVAD(samples []float32) error {
p.totalSamples += len(samples) p.totalSamples += len(samples)
p.transcriber.AcceptVAD(samples) p.transcriber.AcceptVAD(samples)
if p.previewActive || audio.RMSDBFS(samples) >= p.settings.PreviewThresholdDB { speechActive := p.transcriber.SpeechActive()
if speechActive && (p.previewActive || audio.RMSDBFS(samples) >= p.settings.PreviewThresholdDB) {
if !p.previewActive { if !p.previewActive {
p.previewActive = true p.previewActive = true
p.previewStart = p.totalSamples - len(samples) p.previewStart = p.totalSamples - len(samples)
} }
p.previewSamples = append(p.previewSamples, samples...) p.previewSamples = append(p.previewSamples, samples...)
if len(p.previewSamples) >= p.nextPreviewAt { p.previewReceived += len(samples)
p.trimPreviewWindow()
if p.previewReceived >= p.nextPreviewAt {
if err := p.emitProvisional(); err != nil { if err := p.emitProvisional(); err != nil {
return err return err
} }
p.nextPreviewAt += audio.DurationSamples(p.settings.ChunkDuration) p.nextPreviewAt += audio.DurationSamples(p.settings.ChunkDuration)
} }
} }
return p.drainVAD() if err := p.drainVAD(); err != nil {
return err
}
// A short VAD false-positive can become active without meeting the minimum
// speech duration, so no final segment is produced. Clear its provisional
// text instead of leaving a stale caption on screen.
if !speechActive && p.previewActive {
p.resetPreview()
return p.emit(Event{Kind: Hide})
}
return nil
} }
func (p *Processor) drainVAD() error { func (p *Processor) drainVAD() error {
@@ -93,13 +121,28 @@ func (p *Processor) drainVAD() error {
if err := p.emitFinal(start, start+len(samples), samples); err != nil { if err := p.emitFinal(start, start+len(samples), samples); err != nil {
return err return err
} }
p.previewSamples = nil p.resetPreview()
p.previewActive = false
p.previewStart = 0
p.nextPreviewAt = audio.DurationSamples(p.settings.ChunkDuration)
} }
} }
func (p *Processor) trimPreviewWindow() {
maxSamples := audio.DurationSamples(maxPreviewDuration)
if len(p.previewSamples) <= maxSamples {
return
}
dropped := len(p.previewSamples) - maxSamples
p.previewSamples = p.previewSamples[dropped:]
p.previewStart += dropped
}
func (p *Processor) resetPreview() {
p.previewSamples = nil
p.previewActive = false
p.previewStart = 0
p.previewReceived = 0
p.nextPreviewAt = audio.DurationSamples(p.settings.ChunkDuration)
}
func (p *Processor) emitProvisional() error { func (p *Processor) emitProvisional() error {
text := strings.TrimSpace(p.transcriber.Decode(p.previewSamples)) text := strings.TrimSpace(p.transcriber.Decode(p.previewSamples))
if text == "" { if text == "" {
@@ -125,3 +168,10 @@ func (p *Processor) emitFinal(start, end int, samples []float32) error {
EndedAt: audio.SamplesDuration(end), EndedAt: audio.SamplesDuration(end),
}) })
} }
func (p *Processor) emitFixedFinal(start, end int, samples []float32) error {
if audio.RMSDBFS(samples) < p.settings.PreviewThresholdDB {
return p.emit(Event{Kind: Hide})
}
return p.emitFinal(start, end, samples)
}
+130 -18
View File
@@ -14,13 +14,17 @@ type fakeSegment struct {
} }
type fakeTranscriber struct { type fakeTranscriber struct {
text string text string
decodeTexts []string decodeTexts []string
segments []fakeSegment segments []fakeSegment
flushed bool flushed bool
speechActive bool
decodeSizes []int
acceptedVAD int
} }
func (f *fakeTranscriber) Decode([]float32) string { func (f *fakeTranscriber) Decode(samples []float32) string {
f.decodeSizes = append(f.decodeSizes, len(samples))
if len(f.decodeTexts) == 0 { if len(f.decodeTexts) == 0 {
return f.text return f.text
} }
@@ -28,8 +32,9 @@ func (f *fakeTranscriber) Decode([]float32) string {
f.decodeTexts = f.decodeTexts[1:] f.decodeTexts = f.decodeTexts[1:]
return text return text
} }
func (f *fakeTranscriber) AcceptVAD([]float32) {} func (f *fakeTranscriber) AcceptVAD(samples []float32) { f.acceptedVAD += len(samples) }
func (f *fakeTranscriber) FlushVAD() { f.flushed = true } func (f *fakeTranscriber) SpeechActive() bool { return f.speechActive }
func (f *fakeTranscriber) FlushVAD() { f.flushed = true }
func (f *fakeTranscriber) NextSpeechSegment() (int, []float32, bool) { func (f *fakeTranscriber) NextSpeechSegment() (int, []float32, bool) {
if len(f.segments) == 0 { if len(f.segments) == 0 {
return 0, nil, false return 0, nil, false
@@ -42,12 +47,14 @@ func (f *fakeTranscriber) NextSpeechSegment() (int, []float32, bool) {
func TestFixedModeEmitsFinalCaption(t *testing.T) { func TestFixedModeEmitsFinalCaption(t *testing.T) {
transcriber := &fakeTranscriber{text: "hello"} transcriber := &fakeTranscriber{text: "hello"}
var events []Event var events []Event
processor := New(config.Settings{Mode: config.ModeFixed, ChunkDuration: time.Second}, transcriber, func(event Event) error { processor := New(config.Settings{
Mode: config.ModeFixed, ChunkDuration: time.Second, PreviewThresholdDB: -45,
}, transcriber, func(event Event) error {
events = append(events, event) events = append(events, event)
return nil return nil
}) })
if err := processor.Accept(make([]float32, audio.SampleRate)); err != nil { if err := processor.Accept(constantSamples(audio.SampleRate, 0.5)); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if len(events) != 1 || events[0].Kind != Final || events[0].Text != "hello" { if len(events) != 1 || events[0].Kind != Final || events[0].Text != "hello" {
@@ -61,12 +68,14 @@ func TestFixedModeEmitsFinalCaption(t *testing.T) {
func TestFixedModeFlushesFinalPartialChunkWithExactTimestamps(t *testing.T) { func TestFixedModeFlushesFinalPartialChunkWithExactTimestamps(t *testing.T) {
transcriber := &fakeTranscriber{decodeTexts: []string{"first", "partial"}} transcriber := &fakeTranscriber{decodeTexts: []string{"first", "partial"}}
var events []Event var events []Event
processor := New(config.Settings{Mode: config.ModeFixed, ChunkDuration: time.Second}, transcriber, func(event Event) error { processor := New(config.Settings{
Mode: config.ModeFixed, ChunkDuration: time.Second, PreviewThresholdDB: -45,
}, transcriber, func(event Event) error {
events = append(events, event) events = append(events, event)
return nil return nil
}) })
if err := processor.Accept(make([]float32, audio.SampleRate+audio.SampleRate/2)); err != nil { if err := processor.Accept(constantSamples(audio.SampleRate+audio.SampleRate/2, 0.5)); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := processor.Flush(); err != nil { if err := processor.Flush(); err != nil {
@@ -81,7 +90,7 @@ func TestFixedModeFlushesFinalPartialChunkWithExactTimestamps(t *testing.T) {
} }
func TestVADModeEmitsProvisionalThenFinal(t *testing.T) { func TestVADModeEmitsProvisionalThenFinal(t *testing.T) {
transcriber := &fakeTranscriber{text: "speech"} transcriber := &fakeTranscriber{text: "speech", speechActive: true}
var events []Event var events []Event
processor := New(config.Settings{ processor := New(config.Settings{
Mode: config.ModeVAD, Mode: config.ModeVAD,
@@ -92,14 +101,12 @@ func TestVADModeEmitsProvisionalThenFinal(t *testing.T) {
return nil return nil
}) })
loud := make([]float32, audio.SampleRate) loud := constantSamples(audio.SampleRate, 0.5)
for i := range loud {
loud[i] = 0.5
}
if err := processor.Accept(loud); err != nil { if err := processor.Accept(loud); err != nil {
t.Fatal(err) t.Fatal(err)
} }
transcriber.segments = []fakeSegment{{start: 0, samples: loud}} transcriber.segments = []fakeSegment{{start: 0, samples: loud}}
transcriber.speechActive = false
if err := processor.Accept(make([]float32, audio.SamplesPerPacket())); err != nil { if err := processor.Accept(make([]float32, audio.SamplesPerPacket())); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -112,11 +119,13 @@ func TestVADModeEmitsProvisionalThenFinal(t *testing.T) {
func TestEmptyFinalEmitsHide(t *testing.T) { func TestEmptyFinalEmitsHide(t *testing.T) {
transcriber := &fakeTranscriber{text: " \n\t "} transcriber := &fakeTranscriber{text: " \n\t "}
var event Event var event Event
processor := New(config.Settings{Mode: config.ModeFixed, ChunkDuration: time.Second}, transcriber, func(got Event) error { processor := New(config.Settings{
Mode: config.ModeFixed, ChunkDuration: time.Second, PreviewThresholdDB: -45,
}, transcriber, func(got Event) error {
event = got event = got
return nil return nil
}) })
if err := processor.Accept(make([]float32, audio.SampleRate)); err != nil { if err := processor.Accept(constantSamples(audio.SampleRate, 0.5)); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if event.Kind != Hide { if event.Kind != Hide {
@@ -124,6 +133,101 @@ func TestEmptyFinalEmitsHide(t *testing.T) {
} }
} }
func TestFixedModeSilenceStaysOutOfRecognizer(t *testing.T) {
transcriber := &fakeTranscriber{text: "hallucinated caption"}
var events []Event
processor := New(config.Settings{
Mode: config.ModeFixed,
ChunkDuration: time.Second,
PreviewThresholdDB: -45,
}, transcriber, func(event Event) error {
events = append(events, event)
return nil
})
if err := processor.Accept(make([]float32, audio.SampleRate)); err != nil {
t.Fatal(err)
}
if len(transcriber.decodeSizes) != 0 {
t.Fatalf("silent audio reached recognizer: decode sizes = %v", transcriber.decodeSizes)
}
if len(events) != 1 || events[0].Kind != Hide {
t.Fatalf("silent chunk events = %#v, want one Hide", events)
}
}
func TestVADModeDoesNotPreviewLoudNonSpeech(t *testing.T) {
transcriber := &fakeTranscriber{text: "music hallucination"}
var events []Event
processor := New(config.Settings{
Mode: config.ModeVAD,
ChunkDuration: time.Second,
PreviewThresholdDB: -45,
}, transcriber, func(event Event) error {
events = append(events, event)
return nil
})
loudMusic := constantSamples(audio.SampleRate, 0.5)
if err := processor.Accept(loudMusic); err != nil {
t.Fatal(err)
}
if transcriber.acceptedVAD != len(loudMusic) {
t.Fatalf("VAD received %d samples, want %d", transcriber.acceptedVAD, len(loudMusic))
}
if len(transcriber.decodeSizes) != 0 || len(events) != 0 {
t.Fatalf("non-speech reached recognizer: decodes=%v events=%#v", transcriber.decodeSizes, events)
}
}
func TestVADModeClearsRejectedShortSpeechPreview(t *testing.T) {
transcriber := &fakeTranscriber{text: "draft", speechActive: true}
var events []Event
processor := New(config.Settings{
Mode: config.ModeVAD,
ChunkDuration: time.Second,
PreviewThresholdDB: -45,
}, transcriber, func(event Event) error {
events = append(events, event)
return nil
})
if err := processor.Accept(constantSamples(audio.SampleRate, 0.5)); err != nil {
t.Fatal(err)
}
transcriber.speechActive = false
if err := processor.Accept(make([]float32, audio.SamplesPerPacket())); err != nil {
t.Fatal(err)
}
if len(events) != 2 || events[0].Kind != Provisional || events[1].Kind != Hide {
t.Fatalf("events = %#v, want Provisional then Hide", events)
}
}
func TestVADPreviewRecognitionWindowIsBounded(t *testing.T) {
transcriber := &fakeTranscriber{text: "draft", speechActive: true}
processor := New(config.Settings{
Mode: config.ModeVAD,
ChunkDuration: time.Second,
PreviewThresholdDB: -45,
}, transcriber, func(Event) error { return nil })
for range 15 {
if err := processor.Accept(constantSamples(audio.SampleRate, 0.5)); err != nil {
t.Fatal(err)
}
}
maxSamples := audio.DurationSamples(maxPreviewDuration)
for _, size := range transcriber.decodeSizes {
if size > maxSamples {
t.Fatalf("preview decode used %d samples, limit is %d", size, maxSamples)
}
}
if len(transcriber.decodeSizes) != 15 {
t.Fatalf("preview decode count = %d, want 15", len(transcriber.decodeSizes))
}
}
func TestFlushFlushesVAD(t *testing.T) { func TestFlushFlushesVAD(t *testing.T) {
transcriber := &fakeTranscriber{} transcriber := &fakeTranscriber{}
processor := New(config.Settings{Mode: config.ModeVAD, ChunkDuration: time.Second}, transcriber, func(Event) error { return nil }) processor := New(config.Settings{Mode: config.ModeVAD, ChunkDuration: time.Second}, transcriber, func(Event) error { return nil })
@@ -134,3 +238,11 @@ func TestFlushFlushesVAD(t *testing.T) {
t.Fatal("VAD was not flushed") t.Fatal("VAD was not flushed")
} }
} }
func constantSamples(count int, value float32) []float32 {
samples := make([]float32, count)
for i := range samples {
samples[i] = value
}
return samples
}
+2
View File
@@ -10,6 +10,7 @@ import (
"os/exec" "os/exec"
"strings" "strings"
"sync" "sync"
"syscall"
"tea.chunkbyte.com/kato/captioneer/src/audio" "tea.chunkbyte.com/kato/captioneer/src/audio"
) )
@@ -41,6 +42,7 @@ func Packets(ctx context.Context, monitorSource string) (<-chan []float32, func(
"--latency-msec=50", "--latency-msec=50",
) )
cmd.Stderr = os.Stderr cmd.Stderr = os.Stderr
cmd.SysProcAttr = &syscall.SysProcAttr{Pdeathsig: syscall.SIGKILL}
stdout, err := cmd.StdoutPipe() stdout, err := cmd.StdoutPipe()
if err != nil { if err != nil {
return nil, nil, err return nil, nil, err
+4
View File
@@ -20,6 +20,10 @@ import (
) )
func main() { func main() {
if app.IsModelWorker() {
os.Exit(app.RunModelWorker(os.Stderr))
}
runtime.LockOSThread() runtime.LockOSThread()
defer runtime.UnlockOSThread() defer runtime.UnlockOSThread()
+4
View File
@@ -13,6 +13,10 @@ import (
) )
func main() { func main() {
if app.IsModelWorker() {
os.Exit(app.RunModelWorker(os.Stderr))
}
settings := config.Parse() settings := config.Parse()
if err := settings.Validate(); err != nil { if err := settings.Validate(); err != nil {
log.Fatal(err) log.Fatal(err)
+31 -7
View File
@@ -10,8 +10,11 @@ import (
const ( const (
DefaultModelsDir = "models" DefaultModelsDir = "models"
DefaultVADThreshold = 0.5
DefaultOverlayFontFamily = "Sans" DefaultOverlayFontFamily = "Sans"
MinimumOverlayFinalLifetime = 3 * time.Second MinimumOverlayFinalLifetime = 3 * time.Second
DefaultModelTimeout = 30 * time.Second
DefaultModelShutdownTimeout = 2 * time.Second
) )
type Mode string type Mode string
@@ -22,11 +25,15 @@ const (
) )
type Settings struct { type Settings struct {
Mode Mode Mode Mode
ChunkDuration time.Duration ChunkDuration time.Duration
ModelsDir string ModelsDir string
Threads int Threads int
PreviewThresholdDB float64 PreviewThresholdDB float64
VADThreshold float64
ModelAutoRestart bool
ModelTimeout time.Duration
ModelShutdownTimeout time.Duration
} }
func Parse() Settings { func Parse() Settings {
@@ -36,7 +43,11 @@ func Parse() Settings {
flag.DurationVar(&settings.ChunkDuration, "chunk-duration", time.Second, "fixed chunk size or VAD preview refresh interval") flag.DurationVar(&settings.ChunkDuration, "chunk-duration", time.Second, "fixed chunk size or VAD preview refresh interval")
flag.StringVar(&settings.ModelsDir, "models-dir", DefaultModelsDir, "directory containing downloaded Sherpa models") flag.StringVar(&settings.ModelsDir, "models-dir", DefaultModelsDir, "directory containing downloaded Sherpa models")
flag.IntVar(&settings.Threads, "threads", 2, "CPU threads used by recognition and VAD") flag.IntVar(&settings.Threads, "threads", 2, "CPU threads used by recognition and VAD")
flag.Float64Var(&settings.PreviewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS threshold used to begin VAD previews") flag.Float64Var(&settings.PreviewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS gate below which recognition stays idle")
flag.Float64Var(&settings.VADThreshold, "vad-threshold", DefaultVADThreshold, "speech detection sensitivity from 0 to 1; higher rejects more non-speech")
flag.BoolVar(&settings.ModelAutoRestart, "model-auto-restart", true, "restart the isolated model worker after a crash or timeout")
flag.DurationVar(&settings.ModelTimeout, "model-timeout", DefaultModelTimeout, "maximum model startup or processing time; zero disables hang detection")
flag.DurationVar(&settings.ModelShutdownTimeout, "model-shutdown-timeout", DefaultModelShutdownTimeout, "time allowed for model flush and cleanup before force-killing the worker")
flag.Parse() flag.Parse()
settings.Mode = Mode(mode) settings.Mode = Mode(mode)
return settings return settings
@@ -86,7 +97,11 @@ func ParseDesktop() DesktopSettings {
flags.DurationVar(&desktop.ChunkDuration, "chunk-duration", time.Second, "fixed chunk size or VAD preview refresh interval") flags.DurationVar(&desktop.ChunkDuration, "chunk-duration", time.Second, "fixed chunk size or VAD preview refresh interval")
flags.StringVar(&desktop.ModelsDir, "models-dir", DefaultModelsDir, "directory containing downloaded Sherpa models") flags.StringVar(&desktop.ModelsDir, "models-dir", DefaultModelsDir, "directory containing downloaded Sherpa models")
flags.IntVar(&desktop.Threads, "threads", 2, "CPU threads used by recognition and VAD") flags.IntVar(&desktop.Threads, "threads", 2, "CPU threads used by recognition and VAD")
flags.Float64Var(&desktop.PreviewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS threshold used to begin VAD previews") flags.Float64Var(&desktop.PreviewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS gate below which recognition stays idle")
flags.Float64Var(&desktop.VADThreshold, "vad-threshold", DefaultVADThreshold, "speech detection sensitivity from 0 to 1; higher rejects more non-speech")
flags.BoolVar(&desktop.ModelAutoRestart, "model-auto-restart", true, "restart the isolated model worker after a crash or timeout")
flags.DurationVar(&desktop.ModelTimeout, "model-timeout", DefaultModelTimeout, "maximum model startup or processing time; zero disables hang detection")
flags.DurationVar(&desktop.ModelShutdownTimeout, "model-shutdown-timeout", DefaultModelShutdownTimeout, "time allowed for model flush and cleanup before force-killing the worker")
flags.StringVar(&output, "output", "", "caption output: terminal, overlay, or both (required)") flags.StringVar(&output, "output", "", "caption output: terminal, overlay, or both (required)")
flags.StringVar(&backend, "overlay-backend", "auto", "display backend: auto, wayland, or x11") flags.StringVar(&backend, "overlay-backend", "auto", "display backend: auto, wayland, or x11")
flags.Float64Var(&desktop.Overlay.Opacity, "overlay-opacity", 0.90, "caption background opacity from 0 to 1") flags.Float64Var(&desktop.Overlay.Opacity, "overlay-opacity", 0.90, "caption background opacity from 0 to 1")
@@ -114,6 +129,15 @@ func (s Settings) Validate() error {
if s.Threads <= 0 { if s.Threads <= 0 {
return errors.New("--threads must be positive") return errors.New("--threads must be positive")
} }
if s.VADThreshold < 0 || s.VADThreshold > 1 {
return errors.New("--vad-threshold must be between 0 and 1")
}
if s.ModelTimeout < 0 {
return errors.New("--model-timeout cannot be negative")
}
if s.ModelShutdownTimeout <= 0 {
return errors.New("--model-shutdown-timeout must be positive")
}
return nil return nil
} }
+18 -3
View File
@@ -6,7 +6,12 @@ import (
) )
func TestSettingsValidate(t *testing.T) { func TestSettingsValidate(t *testing.T) {
valid := Settings{Mode: ModeFixed, ChunkDuration: time.Second, Threads: 1} valid := Settings{
Mode: ModeFixed,
ChunkDuration: time.Second,
Threads: 1,
ModelShutdownTimeout: DefaultModelShutdownTimeout,
}
if err := valid.Validate(); err != nil { if err := valid.Validate(); err != nil {
t.Fatalf("valid settings failed: %v", err) t.Fatalf("valid settings failed: %v", err)
} }
@@ -17,8 +22,14 @@ func TestSettingsValidate(t *testing.T) {
func validDesktopSettings() DesktopSettings { func validDesktopSettings() DesktopSettings {
return DesktopSettings{ return DesktopSettings{
Settings: Settings{Mode: ModeVAD, ChunkDuration: time.Second, Threads: 2}, Settings: Settings{
Output: OutputOverlay, Mode: ModeVAD,
ChunkDuration: time.Second,
Threads: 2,
ModelTimeout: DefaultModelTimeout,
ModelShutdownTimeout: DefaultModelShutdownTimeout,
},
Output: OutputOverlay,
Overlay: OverlaySettings{ Overlay: OverlaySettings{
Backend: BackendAuto, Opacity: 0.9, FontSize: 28, FontFamily: DefaultOverlayFontFamily, BottomMargin: 100, Backend: BackendAuto, Opacity: 0.9, FontSize: 28, FontFamily: DefaultOverlayFontFamily, BottomMargin: 100,
MaxWidth: 1200, Monitor: "auto", FinalTimeout: 4 * time.Second, ClickThrough: true, MaxWidth: 1200, Monitor: "auto", FinalTimeout: 4 * time.Second, ClickThrough: true,
@@ -46,6 +57,10 @@ func TestDesktopSettingsValidate(t *testing.T) {
{"empty monitor", func(s *DesktopSettings) { s.Overlay.Monitor = "" }}, {"empty monitor", func(s *DesktopSettings) { s.Overlay.Monitor = "" }},
{"negative timeout", func(s *DesktopSettings) { s.Overlay.FinalTimeout = -time.Second }}, {"negative timeout", func(s *DesktopSettings) { s.Overlay.FinalTimeout = -time.Second }},
{"short timeout", func(s *DesktopSettings) { s.Overlay.FinalTimeout = time.Second }}, {"short timeout", func(s *DesktopSettings) { s.Overlay.FinalTimeout = time.Second }},
{"negative model timeout", func(s *DesktopSettings) { s.ModelTimeout = -time.Second }},
{"zero model shutdown timeout", func(s *DesktopSettings) { s.ModelShutdownTimeout = 0 }},
{"negative VAD threshold", func(s *DesktopSettings) { s.VADThreshold = -0.1 }},
{"large VAD threshold", func(s *DesktopSettings) { s.VADThreshold = 1.1 }},
} }
for _, test := range tests { for _, test := range tests {
t.Run(test.name, func(t *testing.T) { t.Run(test.name, func(t *testing.T) {
+12 -3
View File
@@ -25,6 +25,11 @@ type Resources struct {
vad *sherpa.VoiceActivityDetector vad *sherpa.VoiceActivityDetector
} }
func Validate(settings config.Settings) error {
_, err := validatePaths(settings.ModelsDir, settings.Mode)
return err
}
func New(settings config.Settings) (*Resources, error) { func New(settings config.Settings) (*Resources, error) {
paths, err := validatePaths(settings.ModelsDir, settings.Mode) paths, err := validatePaths(settings.ModelsDir, settings.Mode)
if err != nil { if err != nil {
@@ -52,7 +57,7 @@ func New(settings config.Settings) (*Resources, error) {
resources := &Resources{recognizer: recognizer} resources := &Resources{recognizer: recognizer}
if settings.Mode == config.ModeVAD { if settings.Mode == config.ModeVAD {
resources.vad = newVAD(paths.vad, settings.Threads) resources.vad = newVAD(paths.vad, settings.Threads, settings.VADThreshold)
if resources.vad == nil { if resources.vad == nil {
resources.Close() resources.Close()
return nil, fmt.Errorf("create Silero VAD: Sherpa returned nil") return nil, fmt.Errorf("create Silero VAD: Sherpa returned nil")
@@ -87,6 +92,10 @@ func (r *Resources) AcceptVAD(samples []float32) {
r.vad.AcceptWaveform(samples) r.vad.AcceptWaveform(samples)
} }
func (r *Resources) SpeechActive() bool {
return r.vad.IsSpeech()
}
func (r *Resources) FlushVAD() { func (r *Resources) FlushVAD() {
r.vad.Flush() r.vad.Flush()
} }
@@ -127,11 +136,11 @@ func validatePaths(modelsDir string, mode config.Mode) (modelPaths, error) {
return paths, nil return paths, nil
} }
func newVAD(model string, threads int) *sherpa.VoiceActivityDetector { func newVAD(model string, threads int, threshold float64) *sherpa.VoiceActivityDetector {
return sherpa.NewVoiceActivityDetector(&sherpa.VadModelConfig{ return sherpa.NewVoiceActivityDetector(&sherpa.VadModelConfig{
SileroVad: sherpa.SileroVadModelConfig{ SileroVad: sherpa.SileroVadModelConfig{
Model: model, Model: model,
Threshold: 0.5, Threshold: float32(threshold),
// A 500 ms pause ends the current caption line and starts the next one. // A 500 ms pause ends the current caption line and starts the next one.
MinSilenceDuration: 0.5, MinSilenceDuration: 0.5,
MinSpeechDuration: 0.25, MinSpeechDuration: 0.25,