feat(upload): add resumable chunk configuration and file validation
Some checks failed
Build and Publish Docker Image / deploy (push) Failing after 56s
Some checks failed
Build and Publish Docker Image / deploy (push) Failing after 56s
- Add `WARPBOX_RESUMABLE_CHUNK_MODE` and `WARPBOX_RESUMABLE_CHUNK_PATH` environment variables to configure temporary chunk storage. - Implement strict file validation for resuming uploads to ensure selected files match the pending session's metadata. - Add `PLANS.md` to document development stages, roadmap, and API specifications (including batching and resumable flows).
This commit is contained in:
111
PLANS.md
Normal file
111
PLANS.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# Warpbox Plans & Staged Development
|
||||
|
||||
This document captures the staged development history and roadmap for Warpbox. For day-to-day usage,
|
||||
configuration, and deployment, see [README.md](./README.md).
|
||||
|
||||
## Stage 2 — Operator Tools
|
||||
|
||||
- `/admin/login` - token-based admin login.
|
||||
- `/admin` - overview metrics: boxes, files, storage, recent uploads, protected/expired boxes.
|
||||
- `/admin/files` - recent upload table with view and delete actions.
|
||||
- Expired boxes and boxes that have reached their download limit are cleaned on startup and then
|
||||
every `WARPBOX_CLEANUP_EVERY` when `WARPBOX_CLEANUP_ENABLED=true`.
|
||||
- Missing image/video thumbnails are generated in a background worker every `WARPBOX_THUMBNAIL_EVERY`
|
||||
when `WARPBOX_THUMBNAIL_ENABLED=true`.
|
||||
|
||||
## Stage 3 — Anonymous Integrations
|
||||
|
||||
Anonymous uploads return a private management link at creation time. Keep that link secret: anyone
|
||||
with it can delete the entire upload box. The raw delete token is not stored and cannot be recovered
|
||||
later.
|
||||
|
||||
Browser uploads still show `Open box` and `Copy URL` as the primary actions, with a smaller
|
||||
`Manage or delete this upload` link in the completion panel.
|
||||
|
||||
Curl and custom uploaders can use the same endpoint:
|
||||
|
||||
```bash
|
||||
# Terminal-friendly output: one plain box URL.
|
||||
curl -F file=@./report.pdf http://localhost:8080/api/v1/upload
|
||||
|
||||
# JSON output with boxUrl, thumbnailUrl, manageUrl, deleteUrl, zipUrl, and file entries.
|
||||
curl -F sharex=@./screenshot.png \
|
||||
-H 'Accept: application/json' \
|
||||
http://localhost:8080/api/v1/upload
|
||||
```
|
||||
|
||||
The upload endpoint accepts multipart fields named `file` and `sharex`. ShareX users can start from
|
||||
`examples/sharex/warpbox-anonymous.sxcu`; update `RequestURL` to match your instance URL.
|
||||
Authenticated uploads (your account's limits) add an `Authorization: Bearer <token>` header — mint a
|
||||
token under **Account → Access tokens**. The JSON response uses ShareX placeholders `{json:boxUrl}`
|
||||
(URL), `{json:thumbnailUrl}` (thumbnail), `{json:deleteUrl}` (deletion), and `{json:error}` (error
|
||||
message).
|
||||
|
||||
### Grouping multiple files into one box (`X-Warpbox-Batch`)
|
||||
|
||||
By default every uploaded file becomes its own box. To put several files in a **single** box, send
|
||||
the opt-in `X-Warpbox-Batch` header: requests that share the same header value (scoped per account,
|
||||
or per IP for anonymous uploads) within 20s are appended to the same box. This lets a multi-file
|
||||
ShareX selection — which ShareX sends as separate back-to-back requests — land as one shareable link.
|
||||
The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file. Requests
|
||||
without the header behave exactly as before.
|
||||
|
||||
### Resumable API flow
|
||||
|
||||
Custom clients can use the resumable JSON API for large files:
|
||||
|
||||
```bash
|
||||
# 1. Create a resumable session from file metadata.
|
||||
curl -s http://localhost:8080/api/v1/uploads/resumable \
|
||||
-H 'Accept: application/json' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"files":[{"name":"large.bin","size":1048576,"contentType":"application/octet-stream"}],"expiresMinutes":1440}'
|
||||
|
||||
# 2. Upload exact-sized chunks using the returned sessionId, file id, and chunkSize.
|
||||
dd if=./large.bin bs=8388608 count=1 skip=0 2>/dev/null | \
|
||||
curl -X PUT --data-binary @- \
|
||||
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
|
||||
|
||||
# 3. Complete the session after all chunks are present.
|
||||
curl -X POST -H 'Accept: application/json' \
|
||||
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/complete
|
||||
```
|
||||
|
||||
The complete response is the same JSON shape as `POST /api/v1/upload`, including `boxUrl`,
|
||||
`manageUrl`, `deleteUrl`, and file URLs. Send `Authorization: Bearer <token>` on every resumable
|
||||
request to upload as an account.
|
||||
|
||||
Browser resumable uploads are configurable from `/admin/settings`: enabled/disabled, chunk size in
|
||||
MB, draft retention hours, and chunk staging location. The default chunk mode stores temporary draft
|
||||
chunks under `data/tmp/uploads/{session_id}`. A custom mode can point those chunks at another local
|
||||
path, such as a mounted fast SSD. Chunk staging stays local even when the completed files are later
|
||||
finalized into S3/SFTP/SMB/WebDAV storage. Completion returns the share link immediately; files may
|
||||
show as `Processing` on the download page while the background finalizer streams them to the selected
|
||||
storage backend. Cleanup removes expired uploading drafts but skips sessions already in processing.
|
||||
|
||||
## Stage 4 — Accounts + Personal Boxes
|
||||
|
||||
- `/register` bootstraps the first admin account only when no users exist.
|
||||
- `/login` and `/logout` provide cookie-based web sessions.
|
||||
- `/app` is the personal dashboard for logged-in users, showing owned boxes, storage usage, upload
|
||||
history, and flat collections. Uploading still happens from the homepage.
|
||||
- `/admin/users` lets admins create invite links, disable/reactivate users, and generate reset links.
|
||||
- Logged-in browser uploads from `/` still use `POST /api/v1/upload`, but the resulting box is stored
|
||||
with owner and optional collection metadata.
|
||||
- Admin users are exempt from the global max upload size on the homepage upload flow. Future per-user
|
||||
quotas should apply to this same upload path rather than creating a second uploader.
|
||||
- `/admin/settings` controls anonymous uploads, anonymous max upload size, daily upload caps, default
|
||||
user storage quota, and usage retention.
|
||||
- `/admin/users` shows storage/daily usage and lets admins set per-user storage quota overrides.
|
||||
- `/admin/storage` manages the built-in local file backend and S3-compatible bucket backends.
|
||||
- `/admin/bans` manages manual IP/CIDR bans and optional automatic bans for suspicious probes and
|
||||
repeated login failures. Auto-ban is off by default and configured from the admin UI.
|
||||
- Upload limits now include daily bytes, daily box counts, active box counts, short-window request
|
||||
limits, max expiration days, local storage capacity in GB, and per-user policy overrides.
|
||||
- Uploaded file content, thumbnails, and private box metadata use the selected storage backend. The
|
||||
bbolt database and JSON logs remain local under `./data/db` and `./data/logs`.
|
||||
- Anonymous uploads, ShareX uploads, unlisted public box links, password protection, expiry, delete
|
||||
tokens, thumbnails, and cleanup continue to work as before.
|
||||
|
||||
Email delivery is intentionally deferred. Invite and reset links are copyable today; future SMTP
|
||||
support will power public forgot-password and optional email delivery.
|
||||
Reference in New Issue
Block a user