Scrappr

Small Go scraper for the Outward Fandom wiki.

Layout

.
├── cmd/outward-web/main.go    # web UI entrypoint
├── cmd/scrappr/main.go       # binary entrypoint
├── internal/app              # bootstrapping and output writing
├── internal/logx             # colored emoji logger
├── internal/model            # dataset models
├── internal/scraper          # crawl flow, parsing, queueing, retries
├── internal/webui            # embedded web server + static UI
├── go.mod
├── go.sum
└── outward_data.json         # generated output

Run

go run ./cmd/scrappr

go run ./cmd/outward-web

What It Does

• Crawls item and crafting pages from outward.fandom.com
• Uses browser-like headers and rotating user agents
• Limits crawl depth and queue size to avoid drifting into junk pages
• Retries temporary failures with short backoff
• Prints colored emoji logs for queueing, requests, responses, parsing, retries, and periodic status
• Stores legacy and portable infobox fields, primary item image URLs, recipes, effects, and raw content tables for later processing
• Saves resumable checkpoints into .cache/scrape-state.json on a timer, during progress milestones, and on Ctrl+C
• Writes a stable, sorted JSON dataset to outward_data.json
• Serves a local craft-planner UI backed by recipes from outward_data.json

Tuning

Scraper defaults live in internal/scraper/config.go.

• Lower or raise RequestDelay / RequestJitter
• Tighten or relax MaxQueuedPages
• Adjust RequestTimeout, MaxRetries, ProgressEvery, AutosaveEvery, and AutosavePages