warpbox-dev/README.md

# Warpbox.dev

Warpbox is a self-hosted, transfer first file sharing application written in Go. It renders
server side templates and serves static assets directly.

## Features

- Anonymous and authenticated uploads from the browser, `curl`, or ShareX.
- Warpbox-native resumable uploads with a JSON API for large files.
- Upload boxes with expiry, optional download limits, password protection, and one time delete tokens.
- User accounts with personal dashboards, collections, storage quotas, and invite based registration.
- Admin tooling for metrics, file management, storage backends, upload policy, and IP bans.
- Local and S3 compatible storage backends.
- Background jobs for cleanup and thumbnail generation.
- Emoji reaction packs loaded from the runtime data directory.

> Looking for the roadmap and the staged development history? See [PLANS.md](./PLANS.md).

## Run

```bash
./scripts/run/dev.sh
```

The default server listens on `:8080`.

For one off Go commands, run them from the backend module:

```bash
cd backend
go run ./cmd/warpbox
```

## Configuration

All configuration comes from environment variables. The dev script sources `scripts/env/dev.env`.

### Upload size

Upload size limits are configured in megabytes through `WARPBOX_MAX_UPLOAD_SIZE_MB`. Fractions are
supported, so `0.5Mb` is 512 KiB and `1.5Mb` is 1536 KiB.

### Upload policy defaults

These defaults can later be changed from `/admin/settings`:

- `WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true`
- `WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512`
- `WARPBOX_ANONYMOUS_DAILY_UPLOAD_MB=2048`
- `WARPBOX_USER_DAILY_UPLOAD_MB=8192`
- `WARPBOX_DEFAULT_USER_STORAGE_MB=51200`
- `WARPBOX_USAGE_RETENTION_DAYS=30`
- `WARPBOX_LOCAL_STORAGE_MAX_GB=100`
- `WARPBOX_ANONYMOUS_MAX_DAYS=30`
- `WARPBOX_USER_MAX_DAYS=90`
- `WARPBOX_ANONYMOUS_DAILY_BOXES=100`
- `WARPBOX_USER_DAILY_BOXES=250`
- `WARPBOX_ANONYMOUS_ACTIVE_BOXES=500`
- `WARPBOX_USER_ACTIVE_BOXES=1000`
- `WARPBOX_SHORT_WINDOW_REQUESTS=60`
- `WARPBOX_SHORT_WINDOW_SECONDS=60`
- `WARPBOX_ANONYMOUS_STORAGE_BACKEND=local`
- `WARPBOX_USER_STORAGE_BACKEND=local`
- `WARPBOX_RESUMABLE_UPLOADS_ENABLED=true`
- `WARPBOX_RESUMABLE_CHUNK_MB=8`
- `WARPBOX_RESUMABLE_RETENTION_HOURS=24`
- `WARPBOX_RESUMABLE_CHUNK_MODE=same`
- `WARPBOX_RESUMABLE_CHUNK_PATH=`
- `WARPBOX_TRUSTED_PROXIES=` controls whether forwarded client IP headers are accepted only from
 specific proxy IPs/CIDRs. See [SECURITY_PROXY.md](./SECURITY_PROXY.md).

Resumable settings are seeded from the environment and can then be edited from `/admin/settings`.
Saved admin settings override these env defaults. `WARPBOX_RESUMABLE_CHUNK_MODE=same` stores draft
chunks in the normal local temp path, `data/tmp/uploads/{session_id}` under `WARPBOX_DATA_DIR`.
`WARPBOX_RESUMABLE_CHUNK_MODE=custom` uses `WARPBOX_RESUMABLE_CHUNK_PATH` instead, for example a
mounted fast SSD path. Chunk storage is always local temporary staging; completed files are finalized
into the selected storage backend after all chunks arrive.

### Timeouts

Large uploads are expected to take minutes on normal home/server connections. Keep
`WARPBOX_READ_TIMEOUT=0s` and `WARPBOX_WRITE_TIMEOUT=0s` so Go does not close the connection
mid upload; `WARPBOX_READ_HEADER_TIMEOUT=15s` still protects header reads from slowloris style
connections.

### Data directory

Runtime data is configured with `WARPBOX_DATA_DIR` and defaults to `./data` in the dev environment.
The dev script resolves that path from the repository root.

### Background jobs

Background jobs are enabled with `WARPBOX_JOBS_ENABLED=true`. Individual jobs can be toggled with
`WARPBOX_CLEANUP_ENABLED` and `WARPBOX_THUMBNAIL_ENABLED`, and their schedules are configured with
`WARPBOX_CLEANUP_EVERY` and `WARPBOX_THUMBNAIL_EVERY`.

- **Cleanup**: expired boxes and boxes that have reached their download limit are cleaned on startup
 and then on schedule. Stale resumable sessions are removed after `WARPBOX_RESUMABLE_RETENTION_HOURS`.
- **Thumbnails**: missing image/video thumbnails are generated in a background worker.

## First run bootstrap

On a fresh data directory, visit `/register` to create the first account. That first user becomes
the instance admin and normal registration closes after bootstrap. Admins can create copyable invite
links from `/admin/users`.

The env admin token exists as emergency fallback access. Set `WARPBOX_ADMIN_TOKEN` and use it at
`/admin/login` if you need to recover access without a user session.

## Uploads

Browser uploads use Warpbox native resumable uploads by default. Resumable behavior is configurable
from `/admin/settings`, including enable/disable, chunk size, retention, and whether chunks use the
default local temp path or a custom local path such as a fast SSD. When all chunks arrive, Warpbox
returns the share link immediately and marks files as `Processing` until the background finalizer
streams them into the selected storage backend. Draft chunks are deleted once finalization succeeds.
Expired uploading drafts are cleaned after the configured retention window; sessions already in
`Processing` are protected from cleanup while finalization is running.

### Anonymous uploads

Anonymous uploads return a private management link at creation time. Keep that link secret: anyone
with it can delete the entire upload box. The raw delete token is not stored and cannot be recovered
later. Browser uploads show `Open box` and `Copy URL` as the primary actions, with a smaller
`Manage or delete this upload` link in the completion panel.

### `curl` and ShareX

```bash
# Terminal-friendly output: one plain box URL.
curl -F file=@./report.pdf http://localhost:8080/api/v1/upload

# JSON output with boxUrl, thumbnailUrl, manageUrl, deleteUrl, zipUrl, and file entries.
curl -F sharex=@./screenshot.png \
 -H 'Accept: application/json' \
 http://localhost:8080/api/v1/upload
```

The upload endpoint accepts multipart fields named `file` and `sharex`. ShareX users can start from
`examples/sharex/warpbox-anonymous.sxcu`; update `RequestURL` to match your instance URL.
Authenticated uploads (your account's limits) add an `Authorization: Bearer <token>` header mint a
token under **Account → Access tokens**. The JSON response uses ShareX placeholders `{json:boxUrl}`
(URL), `{json:thumbnailUrl}` (thumbnail), `{json:deleteUrl}` (deletion), and `{json:error}` (error
message).

### Grouping multiple files into one box (`X-Warpbox-Batch`)

By default every uploaded file becomes its own box. To put several files in a **single** box, send
the opt-in `X-Warpbox-Batch` header: requests that share the same header value (scoped per account,
or per IP for anonymous uploads) within 20s are appended to the same box. This lets a multi-file
ShareX selection which ShareX sends as separate back-to-back requests land as one shareable link.
The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file. Requests
without the header behave exactly as before.

### Resumable API flow

Custom clients can use the resumable JSON API for large files:

```bash
# 1. Create a resumable session from file metadata.
curl -s http://localhost:8080/api/v1/uploads/resumable \
 -H 'Accept: application/json' \
 -H 'Content-Type: application/json' \
 -d '{"files":[{"name":"large.bin","size":1048576,"contentType":"application/octet-stream"}],"expiresMinutes":1440}'

# 2. Upload exact-sized chunks using the returned sessionId, file id, and chunkSize.
dd if=./large.bin bs=8388608 count=1 skip=0 2>/dev/null | \
 curl -X PUT --data-binary @- \
 http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0

# 3. Complete the session after all chunks are present.
curl -X POST -H 'Accept: application/json' \
 http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/complete
```

The complete response is the same JSON shape as `POST /api/v1/upload`, including `boxUrl`,
`manageUrl`, `deleteUrl`, and file URLs. Send `Authorization: Bearer <token>` on every resumable
request to upload as an account.

## Accounts and admin

- `/register` bootstraps the first admin account only when no users exist.
- `/login` and `/logout` provide cookie-based web sessions.
- `/app` is the personal dashboard for logged-in users, showing owned boxes, storage usage, upload
 history, and flat collections. Uploading still happens from the homepage.
- `/admin` shows overview metrics: boxes, files, storage, recent uploads, protected/expired boxes.
- `/admin/files` is a recent upload table with view and delete actions.
- `/admin/users` lets admins create invite links, disable/reactivate users, generate reset links,
 view storage/daily usage, and set per-user storage quota overrides.
- `/admin/settings` controls anonymous uploads, anonymous max upload size, daily upload caps, default
 user storage quota, and usage retention.
- `/admin/storage` manages the built-in local file backend and S3-compatible bucket backends.
- `/admin/bans` manages manual IP/CIDR bans and optional automatic bans for suspicious probes and
 repeated login failures. Auto-ban is off by default and configured from the admin UI.

Logged-in browser uploads from `/` use `POST /api/v1/upload`, and the resulting box is stored with
owner and optional collection metadata. Admin users are exempt from the global max upload size on the
homepage upload flow.

Email delivery is intentionally deferred. Invite and reset links are copyable today; future SMTP
support will power public forgot-password and optional email delivery.

## Emoji reaction packs

File reactions use emoji packs from the runtime data directory, not from the application code. By
default that means `./data/emoji`; if you change `WARPBOX_DATA_DIR`, use `$WARPBOX_DATA_DIR/emoji`
instead.

Each folder under `./data/emoji` becomes one emoji tab in the reaction picker. Put image files
directly inside the pack folder:

```text
data/
├── db/
├── files/
├── logs/
└── emoji/
  ├── openmoji/
  │  ├── 1F600.svg
  │  ├── 1F44D.svg
  │  └── 2764.svg
  ├── pixel-pack/
  │  ├── happy.webp
  │  ├── fire.webp
  │  └── star.webp
  └── custom-work/
    ├── approved.png
    └── shipped.png
```

In this example, the picker shows tabs named `Openmoji`, `Pixel pack`, and `Custom work`. Supported
emoji image extensions are `.svg`, `.webp`, `.png`, `.jpg`, `.jpeg`, and `.gif`.

## Deployment

### Docker / Podman

Copy the example environment file and adjust values such as `WARPBOX_BASE_URL` and
`WARPBOX_ADMIN_TOKEN` before running the container. Copy the example
[docker-compose.example.yml](./docker-compose.example.yml) to
[docker-compose.yml](./docker-compose.yml), modify as need-be:

```bash
cp .env.example .env
docker compose -f docker-compose.yml up --build
```

The compose example also works with Podman compatible compose tools. Its data volume uses
`./data:/data:Z` for SELinux relabeling, and the container overrides runtime paths to use `/data`,
`/app/static`, and `/app/templates`. The image exposes the health endpoint `/health`, which Docker
and compose healthchecks use.

### Reverse proxy security

Warpbox uses the resolved client IP for anonymous limits, manual bans, and automatic bans. The
default behavior trusts `X-Forwarded-For` and `X-Real-IP` so a normal Caddy reverse proxy works
without extra setup. For hardened deployments where the app port might be reachable from more than one
network, set `WARPBOX_TRUSTED_PROXIES` to trusted proxy IPs/CIDRs. See
[SECURITY_PROXY.md](./SECURITY_PROXY.md) for Caddy examples and Docker/systemd notes.

### Systemd

Build the binary on the server, create a dedicated user, and keep runtime data outside the repo:

```bash
cd /opt/warpbox-dev/backend
go build -o /usr/local/bin/warpbox ./cmd/warpbox
sudo useradd --system --home /var/lib/warpbox --shell /usr/sbin/nologin warpbox
sudo mkdir -p /var/lib/warpbox /etc/warpbox
sudo chown -R warpbox:warpbox /var/lib/warpbox
sudo cp /opt/warpbox-dev/.env.example /etc/warpbox/warpbox.env
```

Example `/etc/warpbox/warpbox.env` values:

```env
WARPBOX_ENV=production
WARPBOX_ADDR=127.0.0.1:6070
WARPBOX_BASE_URL=https://warpbox.dev
WARPBOX_DATA_DIR=/var/lib/warpbox
WARPBOX_STATIC_DIR=/opt/warpbox-dev/backend/static
WARPBOX_TEMPLATE_DIR=/opt/warpbox-dev/backend/templates
WARPBOX_TRUSTED_PROXIES=127.0.0.1,::1
WARPBOX_READ_HEADER_TIMEOUT=15s
WARPBOX_READ_TIMEOUT=0s
WARPBOX_WRITE_TIMEOUT=0s
```

Example `/etc/systemd/system/warpbox.service`:

```ini
[Unit]
Description=Warpbox file sharing service
After=network-online.target
Wants=network-online.target

[Service]
User=warpbox
Group=warpbox
EnvironmentFile=/etc/warpbox/warpbox.env
ExecStart=/usr/local/bin/warpbox
Restart=always
RestartSec=5
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/warpbox

[Install]
WantedBy=multi-user.target
```

Then enable it:

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now warpbox
sudo systemctl status warpbox
```

Put Caddy in front of `127.0.0.1:6070` and keep the Warpbox port closed to the public internet.

## Runtime data

Warpbox keeps local runtime data under the configured data directory:

- `data/files/{box_id}/@each@{file_id}.ext` - uploaded file contents when the local backend is selected.
- `data/files/{box_id}/@thumb@{file_id}.jpg` - generated previews when the local backend is selected.
- `data/tmp/uploads/{session_id}` - temporary local chunks for unfinished resumable uploads when
 the default chunk mode is selected.
- `data/db/warpbox.bbolt` - bbolt metadata database for boxes, file records, users, sessions,
 invites, collections, upload policy settings, daily usage records, manual bans, automatic ban
 settings, abuse counters, and malicious path rules.
- `data/logs/{YYYY-MM-DD}.log` - JSONL logs, one event per line.

Uploaded file content, thumbnails, and private box metadata use the selected storage backend. The
bbolt database and JSON logs always remain local under `./data/db` and `./data/logs`.

## Project layout

- `backend/cmd/warpbox` - main application entry point.
- `backend/libs/config` - environment-backed configuration.
- `backend/libs/httpserver` - server construction and route composition.
- `backend/libs/handlers` - HTTP handlers for pages, API, health, static files.
- `backend/libs/jobs` - background job registration and job loop definitions.
- `backend/libs/middleware` - request logging, recovery, security headers, gzip, request IDs.
- `backend/libs/services` - business logic boundaries.
- `backend/libs/helpers` - small reusable helpers.
- `backend/libs/web` - Go template renderer.
- `backend/templates` - server-rendered Go templates.
- `backend/static/css`, `backend/static/js`, `backend/static/img`, `backend/static/fonts` - public assets served from `/static/`.
- `scripts/run/dev.sh` - local development runner.
- `scripts/env/dev.env.example` - tracked development environment template.
- `scripts/env/dev.env` - local development environment, ignored by git.

## Static asset policy

The static handler sets long-lived immutable caching for images, video, audio, and fonts, shorter
caching for CSS/JS, and gzip compression for compressible responses.

## AI Usage

I have used AI to accelerate development, all of the code has been reviewed by humans. I have mostly used self-hosted models as well as big models from big companies for a monthly subscription fee. 

I have nothing against AI as long as you can tell me what every single line of your code does. That's how I personally view things.