50 lines
1.6 KiB
Markdown
50 lines
1.6 KiB
Markdown
# Scrappr
|
|
|
|
Small Go scraper for the Outward Fandom wiki.
|
|
|
|
## Layout
|
|
|
|
```text
|
|
.
|
|
├── cmd/outward-web/main.go # web UI entrypoint
|
|
├── cmd/scrappr/main.go # binary entrypoint
|
|
├── internal/app # bootstrapping and output writing
|
|
├── internal/logx # colored emoji logger
|
|
├── internal/model # dataset models
|
|
├── internal/scraper # crawl flow, parsing, queueing, retries
|
|
├── internal/webui # embedded web server + static UI
|
|
├── go.mod
|
|
├── go.sum
|
|
└── outward_data.json # generated output
|
|
```
|
|
|
|
## Run
|
|
|
|
```bash
|
|
go run ./cmd/scrappr
|
|
```
|
|
|
|
```bash
|
|
go run ./cmd/outward-web
|
|
```
|
|
|
|
## What It Does
|
|
|
|
- Crawls item and crafting pages from `outward.fandom.com`
|
|
- Uses browser-like headers and rotating user agents
|
|
- Limits crawl depth and queue size to avoid drifting into junk pages
|
|
- Retries temporary failures with short backoff
|
|
- Prints colored emoji logs for queueing, requests, responses, parsing, retries, and periodic status
|
|
- Stores legacy and portable infobox fields, primary item image URLs, recipes, effects, and raw content tables for later processing
|
|
- Saves resumable checkpoints into `.cache/scrape-state.json` on a timer, during progress milestones, and on `Ctrl+C`
|
|
- Writes a stable, sorted JSON dataset to `outward_data.json`
|
|
- Serves a local craft-planner UI backed by recipes from `outward_data.json`
|
|
|
|
## Tuning
|
|
|
|
Scraper defaults live in `internal/scraper/config.go`.
|
|
|
|
- Lower or raise `RequestDelay` / `RequestJitter`
|
|
- Tighten or relax `MaxQueuedPages`
|
|
- Adjust `RequestTimeout`, `MaxRetries`, `ProgressEvery`, `AutosaveEvery`, and `AutosavePages`
|