Daniel Legt
b9d035d71a
Add disk-backed image proxy support to the web UI and expose it via `/api/image`. The proxy validates image URLs, fetches remote images with a timeout, stores image bytes + metadata in a local cache, and serves cached responses with proper content type and cache headers. Also add `SCRAPPR_IMAGE_CACHE` (default `.cache/webui-images`) and pass it through `cmd/outward-web` into `webui.Run`, with startup logging updated to include the cache location. This reduces repeated remote fetches and makes image delivery more reliable for the UI.feat(webui): add cached image proxy with configurable dir Add disk-backed image proxy support to the web UI and expose it via `/api/image`. The proxy validates image URLs, fetches remote images with a timeout, stores image bytes + metadata in a local cache, and serves cached responses with proper content type and cache headers. Also add `SCRAPPR_IMAGE_CACHE` (default `.cache/webui-images`) and pass it through `cmd/outward-web` into `webui.Run`, with startup logging updated to include the cache location. This reduces repeated remote fetches and makes image delivery more reliable for the UI.
Scrappr
Small Go scraper for the Outward Fandom wiki.
Layout
.
├── cmd/outward-web/main.go # web UI entrypoint
├── cmd/scrappr/main.go # binary entrypoint
├── internal/app # bootstrapping and output writing
├── internal/logx # colored emoji logger
├── internal/model # dataset models
├── internal/scraper # crawl flow, parsing, queueing, retries
├── internal/webui # embedded web server + static UI
├── go.mod
├── go.sum
└── outward_data.json # generated output
Run
go run ./cmd/scrappr
go run ./cmd/outward-web
What It Does
- Crawls item and crafting pages from
outward.fandom.com - Uses browser-like headers and rotating user agents
- Limits crawl depth and queue size to avoid drifting into junk pages
- Retries temporary failures with short backoff
- Prints colored emoji logs for queueing, requests, responses, parsing, retries, and periodic status
- Stores legacy and portable infobox fields, primary item image URLs, recipes, effects, and raw content tables for later processing
- Saves resumable checkpoints into
.cache/scrape-state.jsonon a timer, during progress milestones, and onCtrl+C - Writes a stable, sorted JSON dataset to
outward_data.json - Serves a local craft-planner UI backed by recipes from
outward_data.json
Tuning
Scraper defaults live in internal/scraper/config.go.
- Lower or raise
RequestDelay/RequestJitter - Tighten or relax
MaxQueuedPages - Adjust
RequestTimeout,MaxRetries,ProgressEvery,AutosaveEvery, andAutosavePages