feat(uploads): add native resumable upload support
Implement a native chunked resumable upload API and frontend integration to support reliable large file uploads. Changes include: - Added a 3-step resumable upload API flow (create session, upload chunks, complete session). - Introduced configuration options for chunk size, retention hours, and toggling the feature. - Updated the frontend to utilize resumable uploads with progress tracking. - Configured temporary chunk storage under `data/tmp/uploads` with automatic cleanup. - Documented the API flow and configuration in the README.
This commit is contained in:
33
README.md
33
README.md
@@ -33,6 +33,9 @@ Upload policy defaults are also configured in megabytes and can later be changed
|
||||
- `WARPBOX_SHORT_WINDOW_SECONDS=60`
|
||||
- `WARPBOX_ANONYMOUS_STORAGE_BACKEND=local`
|
||||
- `WARPBOX_USER_STORAGE_BACKEND=local`
|
||||
- `WARPBOX_RESUMABLE_UPLOADS_ENABLED=true`
|
||||
- `WARPBOX_RESUMABLE_CHUNK_MB=8`
|
||||
- `WARPBOX_RESUMABLE_RETENTION_HOURS=24`
|
||||
- `WARPBOX_TRUSTED_PROXIES=` controls whether forwarded client IP headers are accepted only from specific proxy IPs/CIDRs. See [SECURITY_PROXY.md](./SECURITY_PROXY.md).
|
||||
|
||||
Runtime data is configured with `WARPBOX_DATA_DIR` and defaults to `./data` in the dev environment.
|
||||
@@ -43,6 +46,10 @@ Large uploads are expected to take minutes on normal home/server connections. Ke
|
||||
mid-upload; `WARPBOX_READ_HEADER_TIMEOUT=15s` still protects header reads from slowloris-style
|
||||
connections.
|
||||
|
||||
Browser uploads use Warpbox-native resumable uploads by default. Chunks are stored temporarily under
|
||||
`data/tmp/uploads/{session_id}` and then streamed into the selected storage backend when the upload
|
||||
is completed. Stale sessions are cleaned by the cleanup job after `WARPBOX_RESUMABLE_RETENTION_HOURS`.
|
||||
|
||||
Background jobs are enabled with `WARPBOX_JOBS_ENABLED=true`. Individual jobs can be toggled with
|
||||
`WARPBOX_CLEANUP_ENABLED` and `WARPBOX_THUMBNAIL_ENABLED`, and their schedules are configured with
|
||||
`WARPBOX_CLEANUP_EVERY` and `WARPBOX_THUMBNAIL_EVERY`.
|
||||
@@ -242,6 +249,31 @@ ShareX selection — which ShareX sends as separate back-to-back requests — la
|
||||
link. The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file.
|
||||
Requests without the header behave exactly as before.
|
||||
|
||||
### Resumable API flow
|
||||
|
||||
Custom clients can use the resumable JSON API for large files:
|
||||
|
||||
```bash
|
||||
# 1. Create a resumable session from file metadata.
|
||||
curl -s http://localhost:8080/api/v1/uploads/resumable \
|
||||
-H 'Accept: application/json' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"files":[{"name":"large.bin","size":1048576,"contentType":"application/octet-stream"}],"expiresMinutes":1440}'
|
||||
|
||||
# 2. Upload exact-sized chunks using the returned sessionId, file id, and chunkSize.
|
||||
dd if=./large.bin bs=8388608 count=1 skip=0 2>/dev/null | \
|
||||
curl -X PUT --data-binary @- \
|
||||
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
|
||||
|
||||
# 3. Complete the session after all chunks are present.
|
||||
curl -X POST -H 'Accept: application/json' \
|
||||
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/complete
|
||||
```
|
||||
|
||||
The complete response is the same JSON shape as `POST /api/v1/upload`, including `boxUrl`,
|
||||
`manageUrl`, `deleteUrl`, and file URLs. Send `Authorization: Bearer <token>` on every resumable
|
||||
request to upload as an account.
|
||||
|
||||
## Stage 4 Accounts + Personal Boxes
|
||||
|
||||
- `/register` bootstraps the first admin account only when no users exist.
|
||||
@@ -275,6 +307,7 @@ Warpbox keeps local runtime data under the configured data directory:
|
||||
|
||||
- `data/files/{box_id}/@each@{file_id}.ext` - uploaded file contents when the local backend is selected.
|
||||
- `data/files/{box_id}/@thumb@{file_id}.jpg` - generated previews when the local backend is selected.
|
||||
- `data/tmp/uploads/{session_id}` - temporary chunks for unfinished resumable uploads.
|
||||
- `data/db/warpbox.bbolt` - bbolt metadata database for boxes and file records.
|
||||
- `data/db/warpbox.bbolt` also stores users, sessions, invites, and collections.
|
||||
- `data/db/warpbox.bbolt` stores upload policy settings and daily usage records keyed by plain IP
|
||||
|
||||
Reference in New Issue
Block a user