kato/warpbox-dev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(upload): add resumable chunk configuration and file validation
Some checks failed
Build and Publish Docker Image / deploy (push) Failing after 56s
Details

Browse Source 
- Add `WARPBOX_RESUMABLE_CHUNK_MODE` and `WARPBOX_RESUMABLE_CHUNK_PATH` environment variables to configure temporary chunk storage.
- Implement strict file validation for resuming uploads to ensure selected files match the pending session's metadata.
- Add `PLANS.md` to document development stages, roadmap, and API specifications (including batching and resumable flows).
This commit is contained in:
Daniel Legt
2026-06-02 22:13:54 +03:00
parent 5cd476e7f3
commit 313c89483c
22 changed files with 1809 additions and 324 deletions
Download Patch File Download Diff File Expand all files Collapse all files

368
README.md
View File

@@ -1,6 +1,20 @@
# Warpbox.dev
This repository contains the Go backend base for `warpbox.dev`, a self-hosted transfer-first file sharing application.
Warpbox is a self-hosted, transfer first file sharing application written in Go. It renders
server side templates and serves static assets directly.
## Features
- Anonymous and authenticated uploads from the browser, `curl`, or ShareX.
- Warpbox-native resumable uploads with a JSON API for large files.
- Upload boxes with expiry, optional download limits, password protection, and one time delete tokens.
- User accounts with personal dashboards, collections, storage quotas, and invite based registration.
- Admin tooling for metrics, file management, storage backends, upload policy, and IP bans.
- Local and S3 compatible storage backends.
- Background jobs for cleanup and thumbnail generation.
- Emoji reaction packs loaded from the runtime data directory.
> Looking for the roadmap and the staged development history? See [PLANS.md](./PLANS.md).
## Run
@@ -10,11 +24,25 @@ This repository contains the Go backend base for `warpbox.dev`, a self-hosted tr
The default server listens on `:8080`.
Upload size limits are configured in megabytes through `WARPBOX_MAX_UPLOAD_SIZE_MB`.
Fractions are supported, so `0.5Mb` is 512 KiB and `1.5Mb` is 1536 KiB.
For one off Go commands, run them from the backend module:
Upload policy defaults are also configured in megabytes and can later be changed from
`/admin/settings`:
```bash
cd backend
go run ./cmd/warpbox
```
## Configuration
All configuration comes from environment variables. The dev script sources `scripts/env/dev.env`.
### Upload size
Upload size limits are configured in megabytes through `WARPBOX_MAX_UPLOAD_SIZE_MB`. Fractions are
supported, so `0.5Mb` is 512 KiB and `1.5Mb` is 1536 KiB.
### Upload policy defaults
These defaults can later be changed from `/admin/settings`:
- `WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true`
- `WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512`
@@ -36,36 +64,147 @@ Upload policy defaults are also configured in megabytes and can later be changed
- `WARPBOX_RESUMABLE_UPLOADS_ENABLED=true`
- `WARPBOX_RESUMABLE_CHUNK_MB=8`
- `WARPBOX_RESUMABLE_RETENTION_HOURS=24`
- `WARPBOX_TRUSTED_PROXIES=` controls whether forwarded client IP headers are accepted only from specific proxy IPs/CIDRs. See [SECURITY_PROXY.md](./SECURITY_PROXY.md).
- `WARPBOX_RESUMABLE_CHUNK_MODE=same`
- `WARPBOX_RESUMABLE_CHUNK_PATH=`
- `WARPBOX_TRUSTED_PROXIES=` controls whether forwarded client IP headers are accepted only from
specific proxy IPs/CIDRs. See [SECURITY_PROXY.md](./SECURITY_PROXY.md).
Resumable settings are seeded from the environment and can then be edited from `/admin/settings`.
Saved admin settings override these env defaults. `WARPBOX_RESUMABLE_CHUNK_MODE=same` stores draft
chunks in the normal local temp path, `data/tmp/uploads/{session_id}` under `WARPBOX_DATA_DIR`.
`WARPBOX_RESUMABLE_CHUNK_MODE=custom` uses `WARPBOX_RESUMABLE_CHUNK_PATH` instead, for example a
mounted fast SSD path. Chunk storage is always local temporary staging; completed files are finalized
into the selected storage backend after all chunks arrive.
### Timeouts
Large uploads are expected to take minutes on normal home/server connections. Keep
`WARPBOX_READ_TIMEOUT=0s` and `WARPBOX_WRITE_TIMEOUT=0s` so Go does not close the connection
mid upload; `WARPBOX_READ_HEADER_TIMEOUT=15s` still protects header reads from slowloris style
connections.
### Data directory
Runtime data is configured with `WARPBOX_DATA_DIR` and defaults to `./data` in the dev environment.
The dev script resolves that path from the repository root.
Large uploads are expected to take minutes on normal home/server connections. Keep
`WARPBOX_READ_TIMEOUT=0s` and `WARPBOX_WRITE_TIMEOUT=0s` so Go does not close the connection
mid-upload; `WARPBOX_READ_HEADER_TIMEOUT=15s` still protects header reads from slowloris-style
connections.
Browser uploads use Warpbox-native resumable uploads by default. Chunks are stored temporarily under
`data/tmp/uploads/{session_id}` and then streamed into the selected storage backend when the upload
is completed. Stale sessions are cleaned by the cleanup job after `WARPBOX_RESUMABLE_RETENTION_HOURS`.
### Background jobs
Background jobs are enabled with `WARPBOX_JOBS_ENABLED=true`. Individual jobs can be toggled with
`WARPBOX_CLEANUP_ENABLED` and `WARPBOX_THUMBNAIL_ENABLED`, and their schedules are configured with
`WARPBOX_CLEANUP_EVERY` and `WARPBOX_THUMBNAIL_EVERY`.
- **Cleanup**: expired boxes and boxes that have reached their download limit are cleaned on startup
and then on schedule. Stale resumable sessions are removed after `WARPBOX_RESUMABLE_RETENTION_HOURS`.
- **Thumbnails**: missing image/video thumbnails are generated in a background worker.
## First run bootstrap
On a fresh data directory, visit `/register` to create the first account. That first user becomes
the instance admin and normal registration closes after bootstrap. Admins can create copyable invite
links from `/admin/users`.
The env admin token still exists as emergency fallback access. Set `WARPBOX_ADMIN_TOKEN` and use it
at `/admin/login` if you need to recover access without a user session.
The env admin token exists as emergency fallback access. Set `WARPBOX_ADMIN_TOKEN` and use it at
`/admin/login` if you need to recover access without a user session.
## Uploads
Browser uploads use Warpbox native resumable uploads by default. Resumable behavior is configurable
from `/admin/settings`, including enable/disable, chunk size, retention, and whether chunks use the
default local temp path or a custom local path such as a fast SSD. When all chunks arrive, Warpbox
returns the share link immediately and marks files as `Processing` until the background finalizer
streams them into the selected storage backend. Draft chunks are deleted once finalization succeeds.
Expired uploading drafts are cleaned after the configured retention window; sessions already in
`Processing` are protected from cleanup while finalization is running.
### Anonymous uploads
Anonymous uploads return a private management link at creation time. Keep that link secret: anyone
with it can delete the entire upload box. The raw delete token is not stored and cannot be recovered
later. Browser uploads show `Open box` and `Copy URL` as the primary actions, with a smaller
`Manage or delete this upload` link in the completion panel.
### `curl` and ShareX
```bash
# Terminal-friendly output: one plain box URL.
curl -F file=@./report.pdf http://localhost:8080/api/v1/upload
# JSON output with boxUrl, thumbnailUrl, manageUrl, deleteUrl, zipUrl, and file entries.
curl -F sharex=@./screenshot.png \
-H 'Accept: application/json' \
http://localhost:8080/api/v1/upload
```
The upload endpoint accepts multipart fields named `file` and `sharex`. ShareX users can start from
`examples/sharex/warpbox-anonymous.sxcu`; update `RequestURL` to match your instance URL.
Authenticated uploads (your account's limits) add an `Authorization: Bearer <token>` header mint a
token under **Account → Access tokens**. The JSON response uses ShareX placeholders `{json:boxUrl}`
(URL), `{json:thumbnailUrl}` (thumbnail), `{json:deleteUrl}` (deletion), and `{json:error}` (error
message).
### Grouping multiple files into one box (`X-Warpbox-Batch`)
By default every uploaded file becomes its own box. To put several files in a **single** box, send
the opt-in `X-Warpbox-Batch` header: requests that share the same header value (scoped per account,
or per IP for anonymous uploads) within 20s are appended to the same box. This lets a multi-file
ShareX selection which ShareX sends as separate back-to-back requests land as one shareable link.
The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file. Requests
without the header behave exactly as before.
### Resumable API flow
Custom clients can use the resumable JSON API for large files:
```bash
# 1. Create a resumable session from file metadata.
curl -s http://localhost:8080/api/v1/uploads/resumable \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"files":[{"name":"large.bin","size":1048576,"contentType":"application/octet-stream"}],"expiresMinutes":1440}'
# 2. Upload exact-sized chunks using the returned sessionId, file id, and chunkSize.
dd if=./large.bin bs=8388608 count=1 skip=0 2>/dev/null | \
curl -X PUT --data-binary @- \
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
# 3. Complete the session after all chunks are present.
curl -X POST -H 'Accept: application/json' \
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/complete
```
The complete response is the same JSON shape as `POST /api/v1/upload`, including `boxUrl`,
`manageUrl`, `deleteUrl`, and file URLs. Send `Authorization: Bearer <token>` on every resumable
request to upload as an account.
## Accounts and admin
- `/register` bootstraps the first admin account only when no users exist.
- `/login` and `/logout` provide cookie-based web sessions.
- `/app` is the personal dashboard for logged-in users, showing owned boxes, storage usage, upload
history, and flat collections. Uploading still happens from the homepage.
- `/admin` shows overview metrics: boxes, files, storage, recent uploads, protected/expired boxes.
- `/admin/files` is a recent upload table with view and delete actions.
- `/admin/users` lets admins create invite links, disable/reactivate users, generate reset links,
view storage/daily usage, and set per-user storage quota overrides.
- `/admin/settings` controls anonymous uploads, anonymous max upload size, daily upload caps, default
user storage quota, and usage retention.
- `/admin/storage` manages the built-in local file backend and S3-compatible bucket backends.
- `/admin/bans` manages manual IP/CIDR bans and optional automatic bans for suspicious probes and
repeated login failures. Auto-ban is off by default and configured from the admin UI.
Logged-in browser uploads from `/` use `POST /api/v1/upload`, and the resulting box is stored with
owner and optional collection metadata. Admin users are exempt from the global max upload size on the
homepage upload flow.
Email delivery is intentionally deferred. Invite and reset links are copyable today; future SMTP
support will power public forgot-password and optional email delivery.
## Emoji reaction packs
File reactions use emoji packs from the runtime data directory, not from the application code. By
default that means `./data/emoji`; if you change `WARPBOX_DATA_DIR`, use
`$WARPBOX_DATA_DIR/emoji` instead.
default that means `./data/emoji`; if you change `WARPBOX_DATA_DIR`, use `$WARPBOX_DATA_DIR/emoji`
instead.
Each folder under `./data/emoji` becomes one emoji tab in the reaction picker. Put image files
directly inside the pack folder:
@@ -76,35 +215,30 @@ data/
├── files/
├── logs/
└── emoji/
├── openmoji/
│ ├── 1F600.svg
│ ├── 1F44D.svg
│ └── 2764.svg
├── pixel-pack/
│ ├── happy.webp
│ ├── fire.webp
│ └── star.webp
└── custom-work/
├── approved.png
└── shipped.png
├── openmoji/
│ ├── 1F600.svg
│ ├── 1F44D.svg
│ └── 2764.svg
├── pixel-pack/
│ ├── happy.webp
│ ├── fire.webp
│ └── star.webp
└── custom-work/
├── approved.png
└── shipped.png
```
In this example, the picker shows tabs named `Openmoji`, `Pixel pack`, and `Custom work`.
Supported emoji image extensions are `.svg`, `.webp`, `.png`, `.jpg`, `.jpeg`, and `.gif`.
In this example, the picker shows tabs named `Openmoji`, `Pixel pack`, and `Custom work`. Supported
emoji image extensions are `.svg`, `.webp`, `.png`, `.jpg`, `.jpeg`, and `.gif`.
For one-off Go commands, run them from the backend module:
## Deployment
```bash
cd backend
go run ./cmd/warpbox
```
## Docker / Podman
### Docker / Podman
Copy the example environment file and adjust values such as `WARPBOX_BASE_URL` and
`WARPBOX_ADMIN_TOKEN` before running the container:
Copy the example [docker-compose.example.yml](./docker-compose.example.yml) to [docker-compose.yml](./docker-compose.yml), modify as need-be
`WARPBOX_ADMIN_TOKEN` before running the container. Copy the example
[docker-compose.example.yml](./docker-compose.example.yml) to
[docker-compose.yml](./docker-compose.yml), modify as need-be:
```bash
cp .env.example .env
@@ -112,20 +246,19 @@ docker compose -f docker-compose.yml up --build
```
The compose example also works with Podman compatible compose tools. Its data volume uses
`./data:/data:Z` for SELinux relabeling, and the container overrides runtime paths to use
`/data`, `/app/static`, and `/app/templates`.
`./data:/data:Z` for SELinux relabeling, and the container overrides runtime paths to use `/data`,
`/app/static`, and `/app/templates`. The image exposes the health endpoint `/health`, which Docker
and compose healthchecks use.
The image exposes the health endpoint: `/health`. Docker and compose healthchecks use it.
## Reverse Proxy Security
### Reverse proxy security
Warpbox uses the resolved client IP for anonymous limits, manual bans, and automatic bans. The
default behavior trusts `X-Forwarded-For` and `X-Real-IP` so a normal Caddy reverse proxy works
without extra setup. For hardened deployments where the app port might be reachable from more than
one network, set `WARPBOX_TRUSTED_PROXIES` to trusted proxy IPs/CIDRs. See
without extra setup. For hardened deployments where the app port might be reachable from more than one
network, set `WARPBOX_TRUSTED_PROXIES` to trusted proxy IPs/CIDRs. See
[SECURITY_PROXY.md](./SECURITY_PROXY.md) for Caddy examples and Docker/systemd notes.
## Systemd
### Systemd
Build the binary on the server, create a dedicated user, and keep runtime data outside the repo:
@@ -187,7 +320,23 @@ sudo systemctl status warpbox
Put Caddy in front of `127.0.0.1:6070` and keep the Warpbox port closed to the public internet.
## Layout
## Runtime data
Warpbox keeps local runtime data under the configured data directory:
- `data/files/{box_id}/@each@{file_id}.ext` - uploaded file contents when the local backend is selected.
- `data/files/{box_id}/@thumb@{file_id}.jpg` - generated previews when the local backend is selected.
- `data/tmp/uploads/{session_id}` - temporary local chunks for unfinished resumable uploads when
the default chunk mode is selected.
- `data/db/warpbox.bbolt` - bbolt metadata database for boxes, file records, users, sessions,
invites, collections, upload policy settings, daily usage records, manual bans, automatic ban
settings, abuse counters, and malicious path rules.
- `data/logs/{YYYY-MM-DD}.log` - JSONL logs, one event per line.
Uploaded file content, thumbnails, and private box metadata use the selected storage backend. The
bbolt database and JSON logs always remain local under `./data/db` and `./data/logs`.
## Project layout
- `backend/cmd/warpbox` - main application entry point.
- `backend/libs/config` - environment-backed configuration.
@@ -195,7 +344,7 @@ Put Caddy in front of `127.0.0.1:6070` and keep the Warpbox port closed to the p
- `backend/libs/handlers` - HTTP handlers for pages, API, health, static files.
- `backend/libs/jobs` - background job registration and job loop definitions.
- `backend/libs/middleware` - request logging, recovery, security headers, gzip, request IDs.
- `backend/libs/services` - business logic boundaries, starting with upload limits.
- `backend/libs/services` - business logic boundaries.
- `backend/libs/helpers` - small reusable helpers.
- `backend/libs/web` - Go template renderer.
- `backend/templates` - server-rendered Go templates.
@@ -204,118 +353,7 @@ Put Caddy in front of `127.0.0.1:6070` and keep the Warpbox port closed to the p
- `scripts/env/dev.env.example` - tracked development environment template.
- `scripts/env/dev.env` - local development environment, ignored by git.
## Stage 2 Operator Tools
## Static asset policy
- `/admin/login` - token-based admin login.
- `/admin` - overview metrics: boxes, files, storage, recent uploads, protected/expired boxes.
- `/admin/files` - recent upload table with view and delete actions.
- Expired boxes and boxes that have reached their download limit are cleaned on startup and then every `WARPBOX_CLEANUP_EVERY` when `WARPBOX_CLEANUP_ENABLED=true`.
- Missing image/video thumbnails are generated in a background worker every `WARPBOX_THUMBNAIL_EVERY` when `WARPBOX_THUMBNAIL_ENABLED=true`.
## Stage 3 Anonymous Integrations
Anonymous uploads now return a private management link at creation time. Keep that link secret:
anyone with it can delete the entire upload box. The raw delete token is not stored and cannot be
recovered later.
Browser uploads still show `Open box` and `Copy URL` as the primary actions, with a smaller
`Manage or delete this upload` link in the completion panel.
Curl and custom uploaders can use the same endpoint:
```bash
# Terminal-friendly output: one plain box URL.
curl -F file=@./report.pdf http://localhost:8080/api/v1/upload
# JSON output with boxUrl, thumbnailUrl, manageUrl, deleteUrl, zipUrl, and file entries.
curl -F sharex=@./screenshot.png \
-H 'Accept: application/json' \
http://localhost:8080/api/v1/upload
```
The upload endpoint accepts multipart fields named `file` and `sharex`. ShareX users can start
from `examples/sharex/warpbox-anonymous.sxcu`; update `RequestURL` to match your instance URL.
Authenticated uploads (your account's limits) add an `Authorization: Bearer <token>` header — mint
a token under **Account → Access tokens**. The JSON response uses ShareX placeholders
`{json:boxUrl}` (URL), `{json:thumbnailUrl}` (thumbnail), `{json:deleteUrl}` (deletion), and
`{json:error}` (error message).
### Grouping multiple files into one box (`X-Warpbox-Batch`)
By default every uploaded file becomes its own box. To put several files in a **single** box, send
the opt-in `X-Warpbox-Batch` header: requests that share the same header value (scoped per account,
or per IP for anonymous uploads) within 20s are appended to the same box. This lets a multi-file
ShareX selection — which ShareX sends as separate back-to-back requests — land as one shareable
link. The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file.
Requests without the header behave exactly as before.
### Resumable API flow
Custom clients can use the resumable JSON API for large files:
```bash
# 1. Create a resumable session from file metadata.
curl -s http://localhost:8080/api/v1/uploads/resumable \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"files":[{"name":"large.bin","size":1048576,"contentType":"application/octet-stream"}],"expiresMinutes":1440}'
# 2. Upload exact-sized chunks using the returned sessionId, file id, and chunkSize.
dd if=./large.bin bs=8388608 count=1 skip=0 2>/dev/null | \
curl -X PUT --data-binary @- \
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
# 3. Complete the session after all chunks are present.
curl -X POST -H 'Accept: application/json' \
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/complete
```
The complete response is the same JSON shape as `POST /api/v1/upload`, including `boxUrl`,
`manageUrl`, `deleteUrl`, and file URLs. Send `Authorization: Bearer <token>` on every resumable
request to upload as an account.
## Stage 4 Accounts + Personal Boxes
- `/register` bootstraps the first admin account only when no users exist.
- `/login` and `/logout` provide cookie-based web sessions.
- `/app` is the personal dashboard for logged-in users, showing owned boxes, storage usage, upload
history, and flat collections. Uploading still happens from the homepage.
- `/admin/users` lets admins create invite links, disable/reactivate users, and generate reset links.
- Logged-in browser uploads from `/` still use `POST /api/v1/upload`, but the resulting box is
stored with owner and optional collection metadata.
- Admin users are exempt from the global max upload size on the homepage upload flow. Future
per-user quotas should apply to this same upload path rather than creating a second uploader.
- `/admin/settings` controls anonymous uploads, anonymous max upload size, daily upload caps, default
user storage quota, and usage retention.
- `/admin/users` shows storage/daily usage and lets admins set per-user storage quota overrides.
- `/admin/storage` manages the built-in local file backend and S3-compatible bucket backends.
- `/admin/bans` manages manual IP/CIDR bans and optional automatic bans for suspicious probes and
repeated login failures. Auto-ban is off by default and configured from the admin UI.
- Upload limits now include daily bytes, daily box counts, active box counts, short-window request
limits, max expiration days, local storage capacity in GB, and per-user policy overrides.
- Uploaded file content, thumbnails, and private box metadata use the selected storage backend.
The bbolt database and JSON logs remain local under `./data/db` and `./data/logs`.
- Anonymous uploads, ShareX uploads, unlisted public box links, password protection, expiry, delete
tokens, thumbnails, and cleanup continue to work as before.
Email delivery is intentionally deferred. Invite and reset links are copyable today; future SMTP
support will power public forgot-password and optional email delivery.
## Runtime Data
Warpbox keeps local runtime data under the configured data directory:
- `data/files/{box_id}/@each@{file_id}.ext` - uploaded file contents when the local backend is selected.
- `data/files/{box_id}/@thumb@{file_id}.jpg` - generated previews when the local backend is selected.
- `data/tmp/uploads/{session_id}` - temporary chunks for unfinished resumable uploads.
- `data/db/warpbox.bbolt` - bbolt metadata database for boxes and file records.
- `data/db/warpbox.bbolt` also stores users, sessions, invites, and collections.
- `data/db/warpbox.bbolt` stores upload policy settings and daily usage records keyed by plain IP
for anonymous uploads and user ID for signed-in uploads.
- `data/db/warpbox.bbolt` stores manual bans, automatic ban settings, abuse counters, and malicious
path rules.
- `data/logs/{YYYY-MM-DD}.log` - JSONL logs, one event per line.
## Static Asset Policy
The static handler sets long-lived immutable caching for images, video, audio, and fonts, shorter caching for CSS/JS, and gzip compression for compressible responses.
The static handler sets long-lived immutable caching for images, video, audio, and fonts, shorter
caching for CSS/JS, and gzip compression for compressible responses.