feat(uploads): add native resumable upload support

Implement a native chunked resumable upload API and frontend integration
to support reliable large file uploads.

Changes include:
- Added a 3-step resumable upload API flow (create session, upload chunks, complete session).
- Introduced configuration options for chunk size, retention hours, and toggling the feature.
- Updated the frontend to utilize resumable uploads with progress tracking.
- Configured temporary chunk storage under `data/tmp/uploads` with automatic cleanup.
- Documented the API flow and configuration in the README.
This commit is contained in:
2026-06-02 17:41:41 +03:00
parent d3b6a86753
commit 5cd476e7f3
16 changed files with 1805 additions and 85 deletions

View File

@@ -33,6 +33,9 @@ Upload policy defaults are also configured in megabytes and can later be changed
- `WARPBOX_SHORT_WINDOW_SECONDS=60`
- `WARPBOX_ANONYMOUS_STORAGE_BACKEND=local`
- `WARPBOX_USER_STORAGE_BACKEND=local`
- `WARPBOX_RESUMABLE_UPLOADS_ENABLED=true`
- `WARPBOX_RESUMABLE_CHUNK_MB=8`
- `WARPBOX_RESUMABLE_RETENTION_HOURS=24`
- `WARPBOX_TRUSTED_PROXIES=` controls whether forwarded client IP headers are accepted only from specific proxy IPs/CIDRs. See [SECURITY_PROXY.md](./SECURITY_PROXY.md).
Runtime data is configured with `WARPBOX_DATA_DIR` and defaults to `./data` in the dev environment.
@@ -43,6 +46,10 @@ Large uploads are expected to take minutes on normal home/server connections. Ke
mid-upload; `WARPBOX_READ_HEADER_TIMEOUT=15s` still protects header reads from slowloris-style
connections.
Browser uploads use Warpbox-native resumable uploads by default. Chunks are stored temporarily under
`data/tmp/uploads/{session_id}` and then streamed into the selected storage backend when the upload
is completed. Stale sessions are cleaned by the cleanup job after `WARPBOX_RESUMABLE_RETENTION_HOURS`.
Background jobs are enabled with `WARPBOX_JOBS_ENABLED=true`. Individual jobs can be toggled with
`WARPBOX_CLEANUP_ENABLED` and `WARPBOX_THUMBNAIL_ENABLED`, and their schedules are configured with
`WARPBOX_CLEANUP_EVERY` and `WARPBOX_THUMBNAIL_EVERY`.
@@ -242,6 +249,31 @@ ShareX selection — which ShareX sends as separate back-to-back requests — la
link. The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file.
Requests without the header behave exactly as before.
### Resumable API flow
Custom clients can use the resumable JSON API for large files:
```bash
# 1. Create a resumable session from file metadata.
curl -s http://localhost:8080/api/v1/uploads/resumable \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"files":[{"name":"large.bin","size":1048576,"contentType":"application/octet-stream"}],"expiresMinutes":1440}'
# 2. Upload exact-sized chunks using the returned sessionId, file id, and chunkSize.
dd if=./large.bin bs=8388608 count=1 skip=0 2>/dev/null | \
curl -X PUT --data-binary @- \
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
# 3. Complete the session after all chunks are present.
curl -X POST -H 'Accept: application/json' \
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/complete
```
The complete response is the same JSON shape as `POST /api/v1/upload`, including `boxUrl`,
`manageUrl`, `deleteUrl`, and file URLs. Send `Authorization: Bearer <token>` on every resumable
request to upload as an account.
## Stage 4 Accounts + Personal Boxes
- `/register` bootstraps the first admin account only when no users exist.
@@ -275,6 +307,7 @@ Warpbox keeps local runtime data under the configured data directory:
- `data/files/{box_id}/@each@{file_id}.ext` - uploaded file contents when the local backend is selected.
- `data/files/{box_id}/@thumb@{file_id}.jpg` - generated previews when the local backend is selected.
- `data/tmp/uploads/{session_id}` - temporary chunks for unfinished resumable uploads.
- `data/db/warpbox.bbolt` - bbolt metadata database for boxes and file records.
- `data/db/warpbox.bbolt` also stores users, sessions, invites, and collections.
- `data/db/warpbox.bbolt` stores upload policy settings and daily usage records keyed by plain IP