refactor: extract flag parsing, validation, and packet processing into functions; add README and .gitignore
This commit is contained in:
@@ -0,0 +1,3 @@
|
|||||||
|
# Downloaded local speech-recognition models.
|
||||||
|
models/parakeet-tdt-v2/
|
||||||
|
models/silero_vad_v5.onnx
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
# audio-listen
|
||||||
|
|
||||||
|
Local, live English captions for the audio playing through your default PulseAudio/PipeWire output device. It captures that output's monitor source, runs the Parakeet TDT v2 model locally, and writes captions to the terminal.
|
||||||
|
|
||||||
|
Everything runs on your machine after the one-time model download. No audio is uploaded anywhere.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Linux on `amd64` / x86_64
|
||||||
|
- Go 1.26.2 or newer
|
||||||
|
- `pactl` and `parec` available on `PATH` (on Debian/Ubuntu: `sudo apt install pulseaudio-utils`)
|
||||||
|
- `curl` and `tar` to download the models
|
||||||
|
- A working PulseAudio-compatible server, including PipeWire's PulseAudio compatibility layer
|
||||||
|
|
||||||
|
The application listens to the monitor of the current default audio sink. Check it with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pactl get-default-sink
|
||||||
|
```
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
Clone the project, then download the required models once:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone <your-repository-url>
|
||||||
|
cd audio-listen
|
||||||
|
./scripts/download-models.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
The download installs the Parakeet TDT v2 int8 model and Silero VAD into `models/`. They are intentionally not committed to Git because they are large.
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
Use VAD mode for normal live captions. It shows a temporary `…` caption while speech is active, then prints a timestamped final caption after 0.5 seconds of silence:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run . --mode=vad
|
||||||
|
```
|
||||||
|
|
||||||
|
Use fixed mode to decode exact time slices, even through silence:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run . --mode=fixed --chunk-duration=1s
|
||||||
|
```
|
||||||
|
|
||||||
|
Build a reusable executable instead of using `go run`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go build -o audio-listen .
|
||||||
|
./audio-listen --mode=vad
|
||||||
|
```
|
||||||
|
|
||||||
|
Press <kbd>Ctrl</kbd>+<kbd>C</kbd> to stop. Pending audio is flushed before exit.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
`--mode` is required. All other options have defaults.
|
||||||
|
|
||||||
|
| Option | Default | Description |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `--mode=vad` | — | Speech-aware captions. This is the recommended everyday mode. |
|
||||||
|
| `--mode=fixed` | — | Decode every fixed-duration audio chunk. |
|
||||||
|
| `--chunk-duration=1s` | `1s` | Fixed chunk length, or VAD preview refresh interval. Go duration syntax is used, e.g. `500ms`, `1s`, `2s`. |
|
||||||
|
| `--models-dir=models` | `models` | Directory containing `parakeet-tdt-v2/` and `silero_vad_v5.onnx`. |
|
||||||
|
| `--threads=2` | `2` | CPU threads used by Parakeet and VAD. Increase only if your CPU has spare capacity. |
|
||||||
|
| `--preview-threshold-dbfs=-45` | `-45` | RMS level that begins a provisional VAD caption. Raise it (for example, `-35`) if background audio starts previews; lower it (for example, `-55`) if quiet speech does not. |
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Faster fixed chunks
|
||||||
|
go run . --mode=fixed --chunk-duration=500ms
|
||||||
|
|
||||||
|
# Fewer VAD preview redraws and four CPU threads
|
||||||
|
go run . --mode=vad --chunk-duration=2s --threads=4
|
||||||
|
|
||||||
|
# Store or reuse models outside this repository
|
||||||
|
go run . --mode=vad --models-dir=/path/to/models
|
||||||
|
```
|
||||||
|
|
||||||
|
There are no environment variables to configure. All supported configuration is supplied with command-line options.
|
||||||
|
|
||||||
|
## Output and troubleshooting
|
||||||
|
|
||||||
|
Final captions go to standard output in this form:
|
||||||
|
|
||||||
|
```text
|
||||||
|
[12.34s-15.67s] This is a completed caption.
|
||||||
|
```
|
||||||
|
|
||||||
|
Startup messages and overload warnings go to standard error, so you can save final captions separately:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go run . --mode=vad > captions.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Missing model files:** run `./scripts/download-models.sh` again.
|
||||||
|
- **`pactl` or `parec` not found:** install `pulseaudio-utils` and start a PulseAudio-compatible audio server.
|
||||||
|
- **No captions while audio plays:** verify the default sink with `pactl get-default-sink`; the app records that sink's `.monitor` source.
|
||||||
|
- **Overload warning:** recognition is behind real time. Try `--threads=4` (or another suitable value) or `--chunk-duration=2s`.
|
||||||
|
|
||||||
|
## Verify changes
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test ./...
|
||||||
|
go build ./...
|
||||||
|
```
|
||||||
+154
@@ -0,0 +1,154 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"math"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
sherpa "github.com/k2-fsa/sherpa-onnx-go-linux"
|
||||||
|
)
|
||||||
|
|
||||||
|
type captionProcessor struct {
|
||||||
|
cfg settings
|
||||||
|
recognizer *sherpa.OfflineRecognizer
|
||||||
|
vad *sherpa.VoiceActivityDetector
|
||||||
|
output io.Writer
|
||||||
|
|
||||||
|
fixedSamples []float32
|
||||||
|
previewSamples []float32
|
||||||
|
previewActive bool
|
||||||
|
previewVisible bool
|
||||||
|
nextPreviewAt int
|
||||||
|
totalSamples int
|
||||||
|
}
|
||||||
|
|
||||||
|
func newCaptionProcessor(cfg settings, recognizer *sherpa.OfflineRecognizer, vad *sherpa.VoiceActivityDetector, output io.Writer) *captionProcessor {
|
||||||
|
return &captionProcessor{
|
||||||
|
cfg: cfg,
|
||||||
|
recognizer: recognizer,
|
||||||
|
vad: vad,
|
||||||
|
output: output,
|
||||||
|
nextPreviewAt: durationSamples(cfg.chunkDuration),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *captionProcessor) accept(samples []float32) {
|
||||||
|
if len(samples) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if p.cfg.mode == "fixed" {
|
||||||
|
p.acceptFixed(samples)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p.acceptVAD(samples)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *captionProcessor) acceptFixed(samples []float32) {
|
||||||
|
p.fixedSamples = append(p.fixedSamples, samples...)
|
||||||
|
chunkSize := durationSamples(p.cfg.chunkDuration)
|
||||||
|
for len(p.fixedSamples) >= chunkSize {
|
||||||
|
p.emitCommitted(p.totalSamples, p.totalSamples+chunkSize, p.fixedSamples[:chunkSize])
|
||||||
|
p.fixedSamples = p.fixedSamples[chunkSize:]
|
||||||
|
p.totalSamples += chunkSize
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *captionProcessor) acceptVAD(samples []float32) {
|
||||||
|
p.totalSamples += len(samples)
|
||||||
|
p.vad.AcceptWaveform(samples)
|
||||||
|
|
||||||
|
if p.previewActive || rmsDBFS(samples) >= p.cfg.previewThresholdDB {
|
||||||
|
p.previewActive = true
|
||||||
|
p.previewSamples = append(p.previewSamples, samples...)
|
||||||
|
if len(p.previewSamples) >= p.nextPreviewAt {
|
||||||
|
p.emitPreview(p.previewSamples)
|
||||||
|
p.nextPreviewAt += durationSamples(p.cfg.chunkDuration)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
p.drainVAD()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *captionProcessor) drainVAD() {
|
||||||
|
for !p.vad.IsEmpty() {
|
||||||
|
segment := p.vad.Front()
|
||||||
|
p.vad.Pop()
|
||||||
|
p.clearPreview()
|
||||||
|
p.emitCommitted(segment.Start, segment.Start+len(segment.Samples), segment.Samples)
|
||||||
|
p.previewSamples = nil
|
||||||
|
p.previewActive = false
|
||||||
|
p.nextPreviewAt = durationSamples(p.cfg.chunkDuration)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *captionProcessor) flush() {
|
||||||
|
if p.cfg.mode == "fixed" {
|
||||||
|
if len(p.fixedSamples) > 0 {
|
||||||
|
p.emitCommitted(p.totalSamples, p.totalSamples+len(p.fixedSamples), p.fixedSamples)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p.vad.Flush()
|
||||||
|
p.drainVAD()
|
||||||
|
p.clearPreview()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *captionProcessor) emitPreview(samples []float32) {
|
||||||
|
text := p.decode(samples)
|
||||||
|
if text == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintf(p.output, "\r\033[2K… %s", text)
|
||||||
|
p.previewVisible = true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *captionProcessor) emitCommitted(start, end int, samples []float32) {
|
||||||
|
text := p.decode(samples)
|
||||||
|
if text == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p.clearPreview()
|
||||||
|
fmt.Fprintf(p.output, "[%s-%s] %s\n", formatAudioTime(start), formatAudioTime(end), text)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *captionProcessor) clearPreview() {
|
||||||
|
if p.previewVisible {
|
||||||
|
fmt.Fprint(p.output, "\r\033[2K")
|
||||||
|
p.previewVisible = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *captionProcessor) decode(samples []float32) string {
|
||||||
|
if len(samples) == 0 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
stream := sherpa.NewOfflineStream(p.recognizer)
|
||||||
|
defer sherpa.DeleteOfflineStream(stream)
|
||||||
|
stream.AcceptWaveform(sampleRate, samples)
|
||||||
|
p.recognizer.Decode(stream)
|
||||||
|
return strings.TrimSpace(stream.GetResult().Text)
|
||||||
|
}
|
||||||
|
|
||||||
|
func durationSamples(duration time.Duration) int {
|
||||||
|
return int(duration * sampleRate / time.Second)
|
||||||
|
}
|
||||||
|
|
||||||
|
func rmsDBFS(samples []float32) float64 {
|
||||||
|
if len(samples) == 0 {
|
||||||
|
return math.Inf(-1)
|
||||||
|
}
|
||||||
|
var sum float64
|
||||||
|
for _, sample := range samples {
|
||||||
|
sum += float64(sample * sample)
|
||||||
|
}
|
||||||
|
rms := math.Sqrt(sum / float64(len(samples)))
|
||||||
|
if rms == 0 {
|
||||||
|
return math.Inf(-1)
|
||||||
|
}
|
||||||
|
return 20 * math.Log10(rms)
|
||||||
|
}
|
||||||
|
|
||||||
|
func formatAudioTime(samples int) string {
|
||||||
|
return fmt.Sprintf("%.2fs", float64(samples)/sampleRate)
|
||||||
|
}
|
||||||
@@ -1,3 +1,5 @@
|
|||||||
module audio-meter
|
module audio-meter
|
||||||
|
|
||||||
go 1.26.2
|
go 1.26.2
|
||||||
|
|
||||||
|
require github.com/k2-fsa/sherpa-onnx-go-linux v1.13.4
|
||||||
|
|||||||
@@ -0,0 +1,2 @@
|
|||||||
|
github.com/k2-fsa/sherpa-onnx-go-linux v1.13.4 h1:FpEUVeTtdkUlb+YToY8rCWFmvB+zT70YKIKdvnDWIzU=
|
||||||
|
github.com/k2-fsa/sherpa-onnx-go-linux v1.13.4/go.mod h1:NXEH2rsBgTdqY59YpPq6CtSBlBAXy/8a9FmpLERU97I=
|
||||||
@@ -4,59 +4,204 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/binary"
|
"encoding/binary"
|
||||||
"errors"
|
"errors"
|
||||||
|
"flag"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"log"
|
"log"
|
||||||
"math"
|
|
||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
"os/signal"
|
"os/signal"
|
||||||
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
sherpa "github.com/k2-fsa/sherpa-onnx-go-linux"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
sampleRate = 48000
|
sampleRate = 16000
|
||||||
channels = 2
|
channels = 1
|
||||||
bytesPerSample = 2 // Signed 16-bit PCM
|
bytesPerSample = 2
|
||||||
windowDuration = 100 * time.Millisecond
|
packetDuration = 100 * time.Millisecond
|
||||||
minimumDBFS = -96.0
|
defaultModelsDir = "models"
|
||||||
meterWidth = 50
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
type settings struct {
|
||||||
|
mode string
|
||||||
|
chunkDuration time.Duration
|
||||||
|
modelsDir string
|
||||||
|
threads int
|
||||||
|
previewThresholdDB float64
|
||||||
|
}
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
for _, program := range []string{"pactl", "parec"} {
|
cfg := parseFlags()
|
||||||
if _, err := exec.LookPath(program); err != nil {
|
if err := validateSettings(cfg); err != nil {
|
||||||
log.Fatalf("%s was not found in PATH: %v", program, err)
|
log.Fatal(err)
|
||||||
}
|
}
|
||||||
|
if err := validatePrograms(); err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
paths, err := validateModels(cfg.modelsDir, cfg.mode)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
ctx, stop := signal.NotifyContext(
|
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||||
context.Background(),
|
|
||||||
os.Interrupt,
|
|
||||||
syscall.SIGTERM,
|
|
||||||
)
|
|
||||||
defer stop()
|
defer stop()
|
||||||
|
|
||||||
defaultSink, err := commandOutput(
|
recognizer := newRecognizer(paths, cfg.threads)
|
||||||
ctx,
|
defer sherpa.DeleteOfflineRecognizer(recognizer)
|
||||||
"pactl",
|
|
||||||
"get-default-sink",
|
var vad *sherpa.VoiceActivityDetector
|
||||||
)
|
if cfg.mode == "vad" {
|
||||||
if err != nil {
|
vad = newVAD(paths.vad, cfg.threads)
|
||||||
log.Fatalf("get default output sink: %v", err)
|
defer sherpa.DeleteVoiceActivityDetector(vad)
|
||||||
}
|
}
|
||||||
|
|
||||||
monitorSource := defaultSink + ".monitor"
|
monitorSource, err := defaultMonitorSource(ctx)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("find default output monitor: %v", err)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(os.Stderr, "Capturing from: %s\n", monitorSource)
|
||||||
|
fmt.Fprintln(os.Stderr, "Press Ctrl+C to stop.")
|
||||||
|
|
||||||
fmt.Printf("Default output: %s\n", defaultSink)
|
packets, waitCapture, err := capturePackets(ctx, monitorSource)
|
||||||
fmt.Printf("Capturing from: %s\n", monitorSource)
|
if err != nil {
|
||||||
fmt.Println("Press Ctrl+C to stop.")
|
log.Fatalf("start system-audio capture: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
cmd := exec.CommandContext(
|
processor := newCaptionProcessor(cfg, recognizer, vad, os.Stdout)
|
||||||
ctx,
|
for packet := range packets {
|
||||||
"parec",
|
processor.accept(packet)
|
||||||
|
}
|
||||||
|
processor.flush()
|
||||||
|
if err := waitCapture(); err != nil && ctx.Err() == nil {
|
||||||
|
log.Fatalf("audio capture stopped unexpectedly: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseFlags() settings {
|
||||||
|
var cfg settings
|
||||||
|
flag.StringVar(&cfg.mode, "mode", "", "caption mode: vad or fixed (required)")
|
||||||
|
flag.DurationVar(&cfg.chunkDuration, "chunk-duration", time.Second, "fixed chunk size or VAD preview refresh interval")
|
||||||
|
flag.StringVar(&cfg.modelsDir, "models-dir", defaultModelsDir, "directory containing downloaded Sherpa models")
|
||||||
|
flag.IntVar(&cfg.threads, "threads", 2, "CPU threads used by recognition and VAD")
|
||||||
|
flag.Float64Var(&cfg.previewThresholdDB, "preview-threshold-dbfs", -45, "RMS dBFS threshold used to begin VAD previews")
|
||||||
|
flag.Parse()
|
||||||
|
return cfg
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateSettings(cfg settings) error {
|
||||||
|
if cfg.mode != "fixed" && cfg.mode != "vad" {
|
||||||
|
return errors.New("--mode is required and must be either fixed or vad")
|
||||||
|
}
|
||||||
|
if cfg.chunkDuration <= 0 {
|
||||||
|
return errors.New("--chunk-duration must be positive")
|
||||||
|
}
|
||||||
|
if cfg.threads <= 0 {
|
||||||
|
return errors.New("--threads must be positive")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validatePrograms() error {
|
||||||
|
for _, program := range []string{"pactl", "parec"} {
|
||||||
|
if _, err := exec.LookPath(program); err != nil {
|
||||||
|
return fmt.Errorf("%s was not found in PATH: %w", program, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type modelPaths struct {
|
||||||
|
encoder string
|
||||||
|
decoder string
|
||||||
|
joiner string
|
||||||
|
tokens string
|
||||||
|
vad string
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateModels(modelsDir, mode string) (modelPaths, error) {
|
||||||
|
parakeetDir := filepath.Join(modelsDir, "parakeet-tdt-v2")
|
||||||
|
paths := modelPaths{
|
||||||
|
encoder: filepath.Join(parakeetDir, "encoder.int8.onnx"),
|
||||||
|
decoder: filepath.Join(parakeetDir, "decoder.int8.onnx"),
|
||||||
|
joiner: filepath.Join(parakeetDir, "joiner.int8.onnx"),
|
||||||
|
tokens: filepath.Join(parakeetDir, "tokens.txt"),
|
||||||
|
vad: filepath.Join(modelsDir, "silero_vad_v5.onnx"),
|
||||||
|
}
|
||||||
|
required := []string{paths.encoder, paths.decoder, paths.joiner, paths.tokens}
|
||||||
|
if mode == "vad" {
|
||||||
|
required = append(required, paths.vad)
|
||||||
|
}
|
||||||
|
var missing []string
|
||||||
|
for _, path := range required {
|
||||||
|
info, err := os.Stat(path)
|
||||||
|
if err != nil || info.IsDir() {
|
||||||
|
missing = append(missing, path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(missing) > 0 {
|
||||||
|
return modelPaths{}, fmt.Errorf("missing required model files:\n %s\nRun ./scripts/download-models.sh", strings.Join(missing, "\n "))
|
||||||
|
}
|
||||||
|
return paths, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newRecognizer(paths modelPaths, threads int) *sherpa.OfflineRecognizer {
|
||||||
|
recognizer := sherpa.NewOfflineRecognizer(&sherpa.OfflineRecognizerConfig{
|
||||||
|
FeatConfig: sherpa.FeatureConfig{SampleRate: sampleRate, FeatureDim: 80},
|
||||||
|
ModelConfig: sherpa.OfflineModelConfig{
|
||||||
|
Transducer: sherpa.OfflineTransducerModelConfig{
|
||||||
|
Encoder: paths.encoder,
|
||||||
|
Decoder: paths.decoder,
|
||||||
|
Joiner: paths.joiner,
|
||||||
|
},
|
||||||
|
Tokens: paths.tokens,
|
||||||
|
NumThreads: threads,
|
||||||
|
Provider: "cpu",
|
||||||
|
ModelType: "nemo_transducer",
|
||||||
|
},
|
||||||
|
DecodingMethod: "greedy_search",
|
||||||
|
})
|
||||||
|
if recognizer == nil {
|
||||||
|
log.Fatal("create Parakeet recognizer: Sherpa returned nil")
|
||||||
|
}
|
||||||
|
return recognizer
|
||||||
|
}
|
||||||
|
|
||||||
|
func newVAD(model string, threads int) *sherpa.VoiceActivityDetector {
|
||||||
|
vad := sherpa.NewVoiceActivityDetector(&sherpa.VadModelConfig{
|
||||||
|
SileroVad: sherpa.SileroVadModelConfig{
|
||||||
|
Model: model,
|
||||||
|
Threshold: 0.5,
|
||||||
|
MinSilenceDuration: 0.5,
|
||||||
|
MinSpeechDuration: 0.25,
|
||||||
|
WindowSize: 512,
|
||||||
|
MaxSpeechDuration: 30,
|
||||||
|
},
|
||||||
|
SampleRate: sampleRate,
|
||||||
|
NumThreads: threads,
|
||||||
|
Provider: "cpu",
|
||||||
|
}, 60)
|
||||||
|
if vad == nil {
|
||||||
|
log.Fatal("create Silero VAD: Sherpa returned nil")
|
||||||
|
}
|
||||||
|
return vad
|
||||||
|
}
|
||||||
|
|
||||||
|
func defaultMonitorSource(ctx context.Context) (string, error) {
|
||||||
|
output, err := exec.CommandContext(ctx, "pactl", "get-default-sink").Output()
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(string(output)) + ".monitor", nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func capturePackets(ctx context.Context, monitorSource string) (<-chan []float32, func() error, error) {
|
||||||
|
cmd := exec.CommandContext(ctx, "parec",
|
||||||
"--device="+monitorSource,
|
"--device="+monitorSource,
|
||||||
"--format=s16le",
|
"--format=s16le",
|
||||||
fmt.Sprintf("--rate=%d", sampleRate),
|
fmt.Sprintf("--rate=%d", sampleRate),
|
||||||
@@ -64,148 +209,62 @@ func main() {
|
|||||||
"--raw",
|
"--raw",
|
||||||
"--latency-msec=50",
|
"--latency-msec=50",
|
||||||
)
|
)
|
||||||
|
|
||||||
cmd.Stderr = os.Stderr
|
cmd.Stderr = os.Stderr
|
||||||
|
|
||||||
stdout, err := cmd.StdoutPipe()
|
stdout, err := cmd.StdoutPipe()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Fatalf("open parec output: %v", err)
|
return nil, nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := cmd.Start(); err != nil {
|
if err := cmd.Start(); err != nil {
|
||||||
log.Fatalf("start parec: %v", err)
|
return nil, nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
framesPerWindow := int(
|
packets := make(chan []float32, 50) // Five seconds at 100 ms per packet.
|
||||||
float64(sampleRate) * windowDuration.Seconds(),
|
var waitOnce sync.Once
|
||||||
)
|
var waitErr error
|
||||||
|
wait := func() error {
|
||||||
buffer := make(
|
waitOnce.Do(func() { waitErr = cmd.Wait() })
|
||||||
[]byte,
|
return waitErr
|
||||||
framesPerWindow*channels*bytesPerSample,
|
}
|
||||||
)
|
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
defer close(packets)
|
||||||
|
packetBytes := make([]byte, samplesPerPacket()*bytesPerSample)
|
||||||
for {
|
for {
|
||||||
n, readErr := io.ReadFull(stdout, buffer)
|
n, readErr := io.ReadFull(stdout, packetBytes)
|
||||||
|
|
||||||
if n > 0 {
|
if n > 0 {
|
||||||
rmsDBFS, peakDBFS := measurePCM16LE(buffer[:n])
|
packet := pcm16LEToFloat32(packetBytes[:n])
|
||||||
|
select {
|
||||||
fmt.Printf(
|
case packets <- packet:
|
||||||
"\rRMS %6.1f dBFS Peak %6.1f dBFS |%s|",
|
default:
|
||||||
rmsDBFS,
|
// Keep the newest audio so captions recover quickly after overload.
|
||||||
peakDBFS,
|
select {
|
||||||
meter(rmsDBFS, -60, 0, meterWidth),
|
case <-packets:
|
||||||
)
|
default:
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case packets <- packet:
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
fmt.Fprintln(os.Stderr, "warning: caption processing overloaded; dropped 100 ms of oldest audio")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if readErr != nil {
|
if readErr != nil {
|
||||||
if ctx.Err() != nil ||
|
if ctx.Err() == nil && !errors.Is(readErr, io.EOF) && !errors.Is(readErr, io.ErrUnexpectedEOF) {
|
||||||
errors.Is(readErr, io.EOF) ||
|
fmt.Fprintf(os.Stderr, "audio capture read error: %v\n", readErr)
|
||||||
errors.Is(readErr, io.ErrUnexpectedEOF) {
|
|
||||||
break
|
|
||||||
}
|
}
|
||||||
|
return
|
||||||
log.Fatalf("read PCM audio: %v", readErr)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}()
|
||||||
fmt.Println()
|
return packets, wait, nil
|
||||||
|
|
||||||
if err := cmd.Wait(); err != nil && ctx.Err() == nil {
|
|
||||||
log.Fatalf("parec stopped unexpectedly: %v", err)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func commandOutput(
|
func samplesPerPacket() int { return int(sampleRate * packetDuration / time.Second) }
|
||||||
ctx context.Context,
|
|
||||||
name string,
|
|
||||||
args ...string,
|
|
||||||
) (string, error) {
|
|
||||||
output, err := exec.CommandContext(
|
|
||||||
ctx,
|
|
||||||
name,
|
|
||||||
args...,
|
|
||||||
).Output()
|
|
||||||
|
|
||||||
if err != nil {
|
func pcm16LEToFloat32(pcm []byte) []float32 {
|
||||||
return "", err
|
samples := make([]float32, len(pcm)/bytesPerSample)
|
||||||
|
for i := range samples {
|
||||||
|
samples[i] = float32(int16(binary.LittleEndian.Uint16(pcm[i*2:]))) / 32768
|
||||||
}
|
}
|
||||||
|
return samples
|
||||||
return strings.TrimSpace(string(output)), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func measurePCM16LE(pcm []byte) (
|
|
||||||
rmsDBFS float64,
|
|
||||||
peakDBFS float64,
|
|
||||||
) {
|
|
||||||
sampleCount := len(pcm) / bytesPerSample
|
|
||||||
|
|
||||||
if sampleCount == 0 {
|
|
||||||
return minimumDBFS, minimumDBFS
|
|
||||||
}
|
|
||||||
|
|
||||||
var sumSquares float64
|
|
||||||
var peak float64
|
|
||||||
|
|
||||||
for offset := 0; offset+1 < len(pcm); offset += bytesPerSample {
|
|
||||||
sample := int16(
|
|
||||||
binary.LittleEndian.Uint16(
|
|
||||||
pcm[offset : offset+2],
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
value := float64(sample) / 32768.0
|
|
||||||
absolute := math.Abs(value)
|
|
||||||
|
|
||||||
sumSquares += value * value
|
|
||||||
|
|
||||||
if absolute > peak {
|
|
||||||
peak = absolute
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
rms := math.Sqrt(
|
|
||||||
sumSquares / float64(sampleCount),
|
|
||||||
)
|
|
||||||
|
|
||||||
return amplitudeToDBFS(rms), amplitudeToDBFS(peak)
|
|
||||||
}
|
|
||||||
|
|
||||||
func amplitudeToDBFS(amplitude float64) float64 {
|
|
||||||
if amplitude <= 0 {
|
|
||||||
return minimumDBFS
|
|
||||||
}
|
|
||||||
|
|
||||||
db := 20 * math.Log10(amplitude)
|
|
||||||
|
|
||||||
if db < minimumDBFS {
|
|
||||||
return minimumDBFS
|
|
||||||
}
|
|
||||||
|
|
||||||
return db
|
|
||||||
}
|
|
||||||
|
|
||||||
func meter(
|
|
||||||
db float64,
|
|
||||||
minimum float64,
|
|
||||||
maximum float64,
|
|
||||||
width int,
|
|
||||||
) string {
|
|
||||||
ratio := (db - minimum) / (maximum - minimum)
|
|
||||||
|
|
||||||
if ratio < 0 {
|
|
||||||
ratio = 0
|
|
||||||
}
|
|
||||||
|
|
||||||
if ratio > 1 {
|
|
||||||
ratio = 1
|
|
||||||
}
|
|
||||||
|
|
||||||
filled := int(
|
|
||||||
math.Round(ratio * float64(width)),
|
|
||||||
)
|
|
||||||
|
|
||||||
return strings.Repeat("#", filled) +
|
|
||||||
strings.Repeat(" ", width-filled)
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,60 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/binary"
|
||||||
|
"math"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestPCM16LEToFloat32(t *testing.T) {
|
||||||
|
pcm := make([]byte, 6)
|
||||||
|
binary.LittleEndian.PutUint16(pcm[0:], uint16(0))
|
||||||
|
binary.LittleEndian.PutUint16(pcm[2:], uint16(32767))
|
||||||
|
binary.LittleEndian.PutUint16(pcm[4:], uint16(32768))
|
||||||
|
got := pcm16LEToFloat32(pcm)
|
||||||
|
want := []float32{0, 32767.0 / 32768.0, -1}
|
||||||
|
for i := range want {
|
||||||
|
if math.Abs(float64(got[i]-want[i])) > 0.00001 {
|
||||||
|
t.Fatalf("sample %d = %f, want %f", i, got[i], want[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDurationSamples(t *testing.T) {
|
||||||
|
if got := durationSamples(time.Second); got != sampleRate {
|
||||||
|
t.Fatalf("one second = %d samples, want %d", got, sampleRate)
|
||||||
|
}
|
||||||
|
if got := durationSamples(1500 * time.Millisecond); got != 24000 {
|
||||||
|
t.Fatalf("1.5 seconds = %d samples, want 24000", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRMSDBFS(t *testing.T) {
|
||||||
|
if got := rmsDBFS([]float32{1, -1}); math.Abs(got) > 0.00001 {
|
||||||
|
t.Fatalf("full-scale RMS = %f dBFS, want 0", got)
|
||||||
|
}
|
||||||
|
if got := rmsDBFS([]float32{0, 0}); !math.IsInf(got, -1) {
|
||||||
|
t.Fatalf("silence = %f dBFS, want -Inf", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateSettings(t *testing.T) {
|
||||||
|
if err := validateSettings(settings{mode: "fixed", chunkDuration: time.Second, threads: 1}); err != nil {
|
||||||
|
t.Fatalf("valid settings failed: %v", err)
|
||||||
|
}
|
||||||
|
if err := validateSettings(settings{mode: "", chunkDuration: time.Second, threads: 1}); err == nil {
|
||||||
|
t.Fatal("missing mode unexpectedly passed")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateModelsExplainsHowToInstallThem(t *testing.T) {
|
||||||
|
_, err := validateModels(t.TempDir(), "vad")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("missing models unexpectedly passed validation")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "./scripts/download-models.sh") {
|
||||||
|
t.Fatalf("missing-model error did not provide setup command: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,5 +0,0 @@
|
|||||||
{
|
|
||||||
"model_type": "nemo-conformer-tdt",
|
|
||||||
"features_size": 128,
|
|
||||||
"subsampling_factor": 8
|
|
||||||
}
|
|
||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
-8193
File diff suppressed because it is too large
Load Diff
Executable
+37
@@ -0,0 +1,37 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
models_dir="$repo_root/models"
|
||||||
|
parakeet_dir="$models_dir/parakeet-tdt-v2"
|
||||||
|
archive_url="https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2"
|
||||||
|
vad_url="https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/silero_vad_v5.onnx"
|
||||||
|
|
||||||
|
command -v curl >/dev/null || { echo "curl is required" >&2; exit 1; }
|
||||||
|
command -v tar >/dev/null || { echo "tar is required" >&2; exit 1; }
|
||||||
|
|
||||||
|
mkdir -p "$models_dir"
|
||||||
|
tmp_dir="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "$tmp_dir"' EXIT
|
||||||
|
|
||||||
|
echo "Downloading Parakeet TDT v2 int8 model..."
|
||||||
|
curl --fail --location --progress-bar "$archive_url" --output "$tmp_dir/parakeet.tar.bz2"
|
||||||
|
tar -xjf "$tmp_dir/parakeet.tar.bz2" -C "$tmp_dir"
|
||||||
|
|
||||||
|
source_dir="$(find "$tmp_dir" -mindepth 1 -maxdepth 1 -type d -name 'sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8' -print -quit)"
|
||||||
|
if [[ -z "$source_dir" ]]; then
|
||||||
|
echo "Downloaded archive did not contain the expected Parakeet directory" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
rm -rf "$parakeet_dir"
|
||||||
|
mkdir -p "$parakeet_dir"
|
||||||
|
cp "$source_dir/encoder.int8.onnx" "$parakeet_dir/"
|
||||||
|
cp "$source_dir/decoder.int8.onnx" "$parakeet_dir/"
|
||||||
|
cp "$source_dir/joiner.int8.onnx" "$parakeet_dir/"
|
||||||
|
cp "$source_dir/tokens.txt" "$parakeet_dir/"
|
||||||
|
|
||||||
|
echo "Downloading Silero VAD..."
|
||||||
|
curl --fail --location --progress-bar "$vad_url" --output "$models_dir/silero_vad_v5.onnx"
|
||||||
|
|
||||||
|
echo "Models installed in $models_dir"
|
||||||
Reference in New Issue
Block a user