2 Commits

Author SHA1 Message Date
313c89483c feat(upload): add resumable chunk configuration and file validation
Some checks failed
Build and Publish Docker Image / deploy (push) Failing after 56s
- Add `WARPBOX_RESUMABLE_CHUNK_MODE` and `WARPBOX_RESUMABLE_CHUNK_PATH` environment variables to configure temporary chunk storage.
- Implement strict file validation for resuming uploads to ensure selected files match the pending session's metadata.
- Add `PLANS.md` to document development stages, roadmap, and API specifications (including batching and resumable flows).
2026-06-02 22:13:54 +03:00
5cd476e7f3 feat(uploads): add native resumable upload support
Implement a native chunked resumable upload API and frontend integration
to support reliable large file uploads.

Changes include:
- Added a 3-step resumable upload API flow (create session, upload chunks, complete session).
- Introduced configuration options for chunk size, retention hours, and toggling the feature.
- Updated the frontend to utilize resumable uploads with progress tracking.
- Configured temporary chunk storage under `data/tmp/uploads` with automatic cleanup.
- Documented the API flow and configuration in the README.
2026-06-02 17:41:41 +03:00
26 changed files with 3445 additions and 240 deletions

View File

@@ -9,6 +9,11 @@ WARPBOX_CLEANUP_ENABLED=true
WARPBOX_CLEANUP_EVERY=1h
WARPBOX_THUMBNAIL_ENABLED=true
WARPBOX_THUMBNAIL_EVERY=1m
WARPBOX_RESUMABLE_UPLOADS_ENABLED=true
WARPBOX_RESUMABLE_CHUNK_MB=64
WARPBOX_RESUMABLE_RETENTION_HOURS=1
WARPBOX_RESUMABLE_CHUNK_MODE=same
WARPBOX_RESUMABLE_CHUNK_PATH=
WARPBOX_MAX_UPLOAD_SIZE_MB=16384
WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true
WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512

111
PLANS.md Normal file
View File

@@ -0,0 +1,111 @@
# Warpbox Plans & Staged Development
This document captures the staged development history and roadmap for Warpbox. For day-to-day usage,
configuration, and deployment, see [README.md](./README.md).
## Stage 2 — Operator Tools
- `/admin/login` - token-based admin login.
- `/admin` - overview metrics: boxes, files, storage, recent uploads, protected/expired boxes.
- `/admin/files` - recent upload table with view and delete actions.
- Expired boxes and boxes that have reached their download limit are cleaned on startup and then
every `WARPBOX_CLEANUP_EVERY` when `WARPBOX_CLEANUP_ENABLED=true`.
- Missing image/video thumbnails are generated in a background worker every `WARPBOX_THUMBNAIL_EVERY`
when `WARPBOX_THUMBNAIL_ENABLED=true`.
## Stage 3 — Anonymous Integrations
Anonymous uploads return a private management link at creation time. Keep that link secret: anyone
with it can delete the entire upload box. The raw delete token is not stored and cannot be recovered
later.
Browser uploads still show `Open box` and `Copy URL` as the primary actions, with a smaller
`Manage or delete this upload` link in the completion panel.
Curl and custom uploaders can use the same endpoint:
```bash
# Terminal-friendly output: one plain box URL.
curl -F file=@./report.pdf http://localhost:8080/api/v1/upload
# JSON output with boxUrl, thumbnailUrl, manageUrl, deleteUrl, zipUrl, and file entries.
curl -F sharex=@./screenshot.png \
-H 'Accept: application/json' \
http://localhost:8080/api/v1/upload
```
The upload endpoint accepts multipart fields named `file` and `sharex`. ShareX users can start from
`examples/sharex/warpbox-anonymous.sxcu`; update `RequestURL` to match your instance URL.
Authenticated uploads (your account's limits) add an `Authorization: Bearer <token>` header — mint a
token under **Account → Access tokens**. The JSON response uses ShareX placeholders `{json:boxUrl}`
(URL), `{json:thumbnailUrl}` (thumbnail), `{json:deleteUrl}` (deletion), and `{json:error}` (error
message).
### Grouping multiple files into one box (`X-Warpbox-Batch`)
By default every uploaded file becomes its own box. To put several files in a **single** box, send
the opt-in `X-Warpbox-Batch` header: requests that share the same header value (scoped per account,
or per IP for anonymous uploads) within 20s are appended to the same box. This lets a multi-file
ShareX selection — which ShareX sends as separate back-to-back requests — land as one shareable link.
The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file. Requests
without the header behave exactly as before.
### Resumable API flow
Custom clients can use the resumable JSON API for large files:
```bash
# 1. Create a resumable session from file metadata.
curl -s http://localhost:8080/api/v1/uploads/resumable \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"files":[{"name":"large.bin","size":1048576,"contentType":"application/octet-stream"}],"expiresMinutes":1440}'
# 2. Upload exact-sized chunks using the returned sessionId, file id, and chunkSize.
dd if=./large.bin bs=8388608 count=1 skip=0 2>/dev/null | \
curl -X PUT --data-binary @- \
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
# 3. Complete the session after all chunks are present.
curl -X POST -H 'Accept: application/json' \
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/complete
```
The complete response is the same JSON shape as `POST /api/v1/upload`, including `boxUrl`,
`manageUrl`, `deleteUrl`, and file URLs. Send `Authorization: Bearer <token>` on every resumable
request to upload as an account.
Browser resumable uploads are configurable from `/admin/settings`: enabled/disabled, chunk size in
MB, draft retention hours, and chunk staging location. The default chunk mode stores temporary draft
chunks under `data/tmp/uploads/{session_id}`. A custom mode can point those chunks at another local
path, such as a mounted fast SSD. Chunk staging stays local even when the completed files are later
finalized into S3/SFTP/SMB/WebDAV storage. Completion returns the share link immediately; files may
show as `Processing` on the download page while the background finalizer streams them to the selected
storage backend. Cleanup removes expired uploading drafts but skips sessions already in processing.
## Stage 4 — Accounts + Personal Boxes
- `/register` bootstraps the first admin account only when no users exist.
- `/login` and `/logout` provide cookie-based web sessions.
- `/app` is the personal dashboard for logged-in users, showing owned boxes, storage usage, upload
history, and flat collections. Uploading still happens from the homepage.
- `/admin/users` lets admins create invite links, disable/reactivate users, and generate reset links.
- Logged-in browser uploads from `/` still use `POST /api/v1/upload`, but the resulting box is stored
with owner and optional collection metadata.
- Admin users are exempt from the global max upload size on the homepage upload flow. Future per-user
quotas should apply to this same upload path rather than creating a second uploader.
- `/admin/settings` controls anonymous uploads, anonymous max upload size, daily upload caps, default
user storage quota, and usage retention.
- `/admin/users` shows storage/daily usage and lets admins set per-user storage quota overrides.
- `/admin/storage` manages the built-in local file backend and S3-compatible bucket backends.
- `/admin/bans` manages manual IP/CIDR bans and optional automatic bans for suspicious probes and
repeated login failures. Auto-ban is off by default and configured from the admin UI.
- Upload limits now include daily bytes, daily box counts, active box counts, short-window request
limits, max expiration days, local storage capacity in GB, and per-user policy overrides.
- Uploaded file content, thumbnails, and private box metadata use the selected storage backend. The
bbolt database and JSON logs remain local under `./data/db` and `./data/logs`.
- Anonymous uploads, ShareX uploads, unlisted public box links, password protection, expiry, delete
tokens, thumbnails, and cleanup continue to work as before.
Email delivery is intentionally deferred. Invite and reset links are copyable today; future SMTP
support will power public forgot-password and optional email delivery.

341
README.md
View File

@@ -1,6 +1,20 @@
# Warpbox.dev
This repository contains the Go backend base for `warpbox.dev`, a self-hosted transfer-first file sharing application.
Warpbox is a self-hosted, transfer first file sharing application written in Go. It renders
server side templates and serves static assets directly.
## Features
- Anonymous and authenticated uploads from the browser, `curl`, or ShareX.
- Warpbox-native resumable uploads with a JSON API for large files.
- Upload boxes with expiry, optional download limits, password protection, and one time delete tokens.
- User accounts with personal dashboards, collections, storage quotas, and invite based registration.
- Admin tooling for metrics, file management, storage backends, upload policy, and IP bans.
- Local and S3 compatible storage backends.
- Background jobs for cleanup and thumbnail generation.
- Emoji reaction packs loaded from the runtime data directory.
> Looking for the roadmap and the staged development history? See [PLANS.md](./PLANS.md).
## Run
@@ -10,11 +24,25 @@ This repository contains the Go backend base for `warpbox.dev`, a self-hosted tr
The default server listens on `:8080`.
Upload size limits are configured in megabytes through `WARPBOX_MAX_UPLOAD_SIZE_MB`.
Fractions are supported, so `0.5Mb` is 512 KiB and `1.5Mb` is 1536 KiB.
For one off Go commands, run them from the backend module:
Upload policy defaults are also configured in megabytes and can later be changed from
`/admin/settings`:
```bash
cd backend
go run ./cmd/warpbox
```
## Configuration
All configuration comes from environment variables. The dev script sources `scripts/env/dev.env`.
### Upload size
Upload size limits are configured in megabytes through `WARPBOX_MAX_UPLOAD_SIZE_MB`. Fractions are
supported, so `0.5Mb` is 512 KiB and `1.5Mb` is 1536 KiB.
### Upload policy defaults
These defaults can later be changed from `/admin/settings`:
- `WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true`
- `WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512`
@@ -33,32 +61,150 @@ Upload policy defaults are also configured in megabytes and can later be changed
- `WARPBOX_SHORT_WINDOW_SECONDS=60`
- `WARPBOX_ANONYMOUS_STORAGE_BACKEND=local`
- `WARPBOX_USER_STORAGE_BACKEND=local`
- `WARPBOX_TRUSTED_PROXIES=` controls whether forwarded client IP headers are accepted only from specific proxy IPs/CIDRs. See [SECURITY_PROXY.md](./SECURITY_PROXY.md).
- `WARPBOX_RESUMABLE_UPLOADS_ENABLED=true`
- `WARPBOX_RESUMABLE_CHUNK_MB=8`
- `WARPBOX_RESUMABLE_RETENTION_HOURS=24`
- `WARPBOX_RESUMABLE_CHUNK_MODE=same`
- `WARPBOX_RESUMABLE_CHUNK_PATH=`
- `WARPBOX_TRUSTED_PROXIES=` controls whether forwarded client IP headers are accepted only from
specific proxy IPs/CIDRs. See [SECURITY_PROXY.md](./SECURITY_PROXY.md).
Resumable settings are seeded from the environment and can then be edited from `/admin/settings`.
Saved admin settings override these env defaults. `WARPBOX_RESUMABLE_CHUNK_MODE=same` stores draft
chunks in the normal local temp path, `data/tmp/uploads/{session_id}` under `WARPBOX_DATA_DIR`.
`WARPBOX_RESUMABLE_CHUNK_MODE=custom` uses `WARPBOX_RESUMABLE_CHUNK_PATH` instead, for example a
mounted fast SSD path. Chunk storage is always local temporary staging; completed files are finalized
into the selected storage backend after all chunks arrive.
### Timeouts
Large uploads are expected to take minutes on normal home/server connections. Keep
`WARPBOX_READ_TIMEOUT=0s` and `WARPBOX_WRITE_TIMEOUT=0s` so Go does not close the connection
mid upload; `WARPBOX_READ_HEADER_TIMEOUT=15s` still protects header reads from slowloris style
connections.
### Data directory
Runtime data is configured with `WARPBOX_DATA_DIR` and defaults to `./data` in the dev environment.
The dev script resolves that path from the repository root.
Large uploads are expected to take minutes on normal home/server connections. Keep
`WARPBOX_READ_TIMEOUT=0s` and `WARPBOX_WRITE_TIMEOUT=0s` so Go does not close the connection
mid-upload; `WARPBOX_READ_HEADER_TIMEOUT=15s` still protects header reads from slowloris-style
connections.
### Background jobs
Background jobs are enabled with `WARPBOX_JOBS_ENABLED=true`. Individual jobs can be toggled with
`WARPBOX_CLEANUP_ENABLED` and `WARPBOX_THUMBNAIL_ENABLED`, and their schedules are configured with
`WARPBOX_CLEANUP_EVERY` and `WARPBOX_THUMBNAIL_EVERY`.
- **Cleanup**: expired boxes and boxes that have reached their download limit are cleaned on startup
and then on schedule. Stale resumable sessions are removed after `WARPBOX_RESUMABLE_RETENTION_HOURS`.
- **Thumbnails**: missing image/video thumbnails are generated in a background worker.
## First run bootstrap
On a fresh data directory, visit `/register` to create the first account. That first user becomes
the instance admin and normal registration closes after bootstrap. Admins can create copyable invite
links from `/admin/users`.
The env admin token still exists as emergency fallback access. Set `WARPBOX_ADMIN_TOKEN` and use it
at `/admin/login` if you need to recover access without a user session.
The env admin token exists as emergency fallback access. Set `WARPBOX_ADMIN_TOKEN` and use it at
`/admin/login` if you need to recover access without a user session.
## Uploads
Browser uploads use Warpbox native resumable uploads by default. Resumable behavior is configurable
from `/admin/settings`, including enable/disable, chunk size, retention, and whether chunks use the
default local temp path or a custom local path such as a fast SSD. When all chunks arrive, Warpbox
returns the share link immediately and marks files as `Processing` until the background finalizer
streams them into the selected storage backend. Draft chunks are deleted once finalization succeeds.
Expired uploading drafts are cleaned after the configured retention window; sessions already in
`Processing` are protected from cleanup while finalization is running.
### Anonymous uploads
Anonymous uploads return a private management link at creation time. Keep that link secret: anyone
with it can delete the entire upload box. The raw delete token is not stored and cannot be recovered
later. Browser uploads show `Open box` and `Copy URL` as the primary actions, with a smaller
`Manage or delete this upload` link in the completion panel.
### `curl` and ShareX
```bash
# Terminal-friendly output: one plain box URL.
curl -F file=@./report.pdf http://localhost:8080/api/v1/upload
# JSON output with boxUrl, thumbnailUrl, manageUrl, deleteUrl, zipUrl, and file entries.
curl -F sharex=@./screenshot.png \
-H 'Accept: application/json' \
http://localhost:8080/api/v1/upload
```
The upload endpoint accepts multipart fields named `file` and `sharex`. ShareX users can start from
`examples/sharex/warpbox-anonymous.sxcu`; update `RequestURL` to match your instance URL.
Authenticated uploads (your account's limits) add an `Authorization: Bearer <token>` header mint a
token under **Account → Access tokens**. The JSON response uses ShareX placeholders `{json:boxUrl}`
(URL), `{json:thumbnailUrl}` (thumbnail), `{json:deleteUrl}` (deletion), and `{json:error}` (error
message).
### Grouping multiple files into one box (`X-Warpbox-Batch`)
By default every uploaded file becomes its own box. To put several files in a **single** box, send
the opt-in `X-Warpbox-Batch` header: requests that share the same header value (scoped per account,
or per IP for anonymous uploads) within 20s are appended to the same box. This lets a multi-file
ShareX selection which ShareX sends as separate back-to-back requests land as one shareable link.
The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file. Requests
without the header behave exactly as before.
### Resumable API flow
Custom clients can use the resumable JSON API for large files:
```bash
# 1. Create a resumable session from file metadata.
curl -s http://localhost:8080/api/v1/uploads/resumable \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"files":[{"name":"large.bin","size":1048576,"contentType":"application/octet-stream"}],"expiresMinutes":1440}'
# 2. Upload exact-sized chunks using the returned sessionId, file id, and chunkSize.
dd if=./large.bin bs=8388608 count=1 skip=0 2>/dev/null | \
curl -X PUT --data-binary @- \
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
# 3. Complete the session after all chunks are present.
curl -X POST -H 'Accept: application/json' \
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/complete
```
The complete response is the same JSON shape as `POST /api/v1/upload`, including `boxUrl`,
`manageUrl`, `deleteUrl`, and file URLs. Send `Authorization: Bearer <token>` on every resumable
request to upload as an account.
## Accounts and admin
- `/register` bootstraps the first admin account only when no users exist.
- `/login` and `/logout` provide cookie-based web sessions.
- `/app` is the personal dashboard for logged-in users, showing owned boxes, storage usage, upload
history, and flat collections. Uploading still happens from the homepage.
- `/admin` shows overview metrics: boxes, files, storage, recent uploads, protected/expired boxes.
- `/admin/files` is a recent upload table with view and delete actions.
- `/admin/users` lets admins create invite links, disable/reactivate users, generate reset links,
view storage/daily usage, and set per-user storage quota overrides.
- `/admin/settings` controls anonymous uploads, anonymous max upload size, daily upload caps, default
user storage quota, and usage retention.
- `/admin/storage` manages the built-in local file backend and S3-compatible bucket backends.
- `/admin/bans` manages manual IP/CIDR bans and optional automatic bans for suspicious probes and
repeated login failures. Auto-ban is off by default and configured from the admin UI.
Logged-in browser uploads from `/` use `POST /api/v1/upload`, and the resulting box is stored with
owner and optional collection metadata. Admin users are exempt from the global max upload size on the
homepage upload flow.
Email delivery is intentionally deferred. Invite and reset links are copyable today; future SMTP
support will power public forgot-password and optional email delivery.
## Emoji reaction packs
File reactions use emoji packs from the runtime data directory, not from the application code. By
default that means `./data/emoji`; if you change `WARPBOX_DATA_DIR`, use
`$WARPBOX_DATA_DIR/emoji` instead.
default that means `./data/emoji`; if you change `WARPBOX_DATA_DIR`, use `$WARPBOX_DATA_DIR/emoji`
instead.
Each folder under `./data/emoji` becomes one emoji tab in the reaction picker. Put image files
directly inside the pack folder:
@@ -69,35 +215,30 @@ data/
├── files/
├── logs/
└── emoji/
├── openmoji/
├── 1F600.svg
├── 1F44D.svg
└── 2764.svg
├── pixel-pack/
├── happy.webp
├── fire.webp
└── star.webp
└── custom-work/
├── approved.png
└── shipped.png
├── openmoji/
│ ├── 1F600.svg
│ ├── 1F44D.svg
│ └── 2764.svg
├── pixel-pack/
│ ├── happy.webp
│ ├── fire.webp
│ └── star.webp
└── custom-work/
├── approved.png
└── shipped.png
```
In this example, the picker shows tabs named `Openmoji`, `Pixel pack`, and `Custom work`.
Supported emoji image extensions are `.svg`, `.webp`, `.png`, `.jpg`, `.jpeg`, and `.gif`.
In this example, the picker shows tabs named `Openmoji`, `Pixel pack`, and `Custom work`. Supported
emoji image extensions are `.svg`, `.webp`, `.png`, `.jpg`, `.jpeg`, and `.gif`.
For one-off Go commands, run them from the backend module:
## Deployment
```bash
cd backend
go run ./cmd/warpbox
```
## Docker / Podman
### Docker / Podman
Copy the example environment file and adjust values such as `WARPBOX_BASE_URL` and
`WARPBOX_ADMIN_TOKEN` before running the container:
Copy the example [docker-compose.example.yml](./docker-compose.example.yml) to [docker-compose.yml](./docker-compose.yml), modify as need-be
`WARPBOX_ADMIN_TOKEN` before running the container. Copy the example
[docker-compose.example.yml](./docker-compose.example.yml) to
[docker-compose.yml](./docker-compose.yml), modify as need-be:
```bash
cp .env.example .env
@@ -105,20 +246,19 @@ docker compose -f docker-compose.yml up --build
```
The compose example also works with Podman compatible compose tools. Its data volume uses
`./data:/data:Z` for SELinux relabeling, and the container overrides runtime paths to use
`/data`, `/app/static`, and `/app/templates`.
`./data:/data:Z` for SELinux relabeling, and the container overrides runtime paths to use `/data`,
`/app/static`, and `/app/templates`. The image exposes the health endpoint `/health`, which Docker
and compose healthchecks use.
The image exposes the health endpoint: `/health`. Docker and compose healthchecks use it.
## Reverse Proxy Security
### Reverse proxy security
Warpbox uses the resolved client IP for anonymous limits, manual bans, and automatic bans. The
default behavior trusts `X-Forwarded-For` and `X-Real-IP` so a normal Caddy reverse proxy works
without extra setup. For hardened deployments where the app port might be reachable from more than
one network, set `WARPBOX_TRUSTED_PROXIES` to trusted proxy IPs/CIDRs. See
without extra setup. For hardened deployments where the app port might be reachable from more than one
network, set `WARPBOX_TRUSTED_PROXIES` to trusted proxy IPs/CIDRs. See
[SECURITY_PROXY.md](./SECURITY_PROXY.md) for Caddy examples and Docker/systemd notes.
## Systemd
### Systemd
Build the binary on the server, create a dedicated user, and keep runtime data outside the repo:
@@ -180,7 +320,23 @@ sudo systemctl status warpbox
Put Caddy in front of `127.0.0.1:6070` and keep the Warpbox port closed to the public internet.
## Layout
## Runtime data
Warpbox keeps local runtime data under the configured data directory:
- `data/files/{box_id}/@each@{file_id}.ext` - uploaded file contents when the local backend is selected.
- `data/files/{box_id}/@thumb@{file_id}.jpg` - generated previews when the local backend is selected.
- `data/tmp/uploads/{session_id}` - temporary local chunks for unfinished resumable uploads when
the default chunk mode is selected.
- `data/db/warpbox.bbolt` - bbolt metadata database for boxes, file records, users, sessions,
invites, collections, upload policy settings, daily usage records, manual bans, automatic ban
settings, abuse counters, and malicious path rules.
- `data/logs/{YYYY-MM-DD}.log` - JSONL logs, one event per line.
Uploaded file content, thumbnails, and private box metadata use the selected storage backend. The
bbolt database and JSON logs always remain local under `./data/db` and `./data/logs`.
## Project layout
- `backend/cmd/warpbox` - main application entry point.
- `backend/libs/config` - environment-backed configuration.
@@ -188,7 +344,7 @@ Put Caddy in front of `127.0.0.1:6070` and keep the Warpbox port closed to the p
- `backend/libs/handlers` - HTTP handlers for pages, API, health, static files.
- `backend/libs/jobs` - background job registration and job loop definitions.
- `backend/libs/middleware` - request logging, recovery, security headers, gzip, request IDs.
- `backend/libs/services` - business logic boundaries, starting with upload limits.
- `backend/libs/services` - business logic boundaries.
- `backend/libs/helpers` - small reusable helpers.
- `backend/libs/web` - Go template renderer.
- `backend/templates` - server-rendered Go templates.
@@ -197,92 +353,7 @@ Put Caddy in front of `127.0.0.1:6070` and keep the Warpbox port closed to the p
- `scripts/env/dev.env.example` - tracked development environment template.
- `scripts/env/dev.env` - local development environment, ignored by git.
## Stage 2 Operator Tools
## Static asset policy
- `/admin/login` - token-based admin login.
- `/admin` - overview metrics: boxes, files, storage, recent uploads, protected/expired boxes.
- `/admin/files` - recent upload table with view and delete actions.
- Expired boxes and boxes that have reached their download limit are cleaned on startup and then every `WARPBOX_CLEANUP_EVERY` when `WARPBOX_CLEANUP_ENABLED=true`.
- Missing image/video thumbnails are generated in a background worker every `WARPBOX_THUMBNAIL_EVERY` when `WARPBOX_THUMBNAIL_ENABLED=true`.
## Stage 3 Anonymous Integrations
Anonymous uploads now return a private management link at creation time. Keep that link secret:
anyone with it can delete the entire upload box. The raw delete token is not stored and cannot be
recovered later.
Browser uploads still show `Open box` and `Copy URL` as the primary actions, with a smaller
`Manage or delete this upload` link in the completion panel.
Curl and custom uploaders can use the same endpoint:
```bash
# Terminal-friendly output: one plain box URL.
curl -F file=@./report.pdf http://localhost:8080/api/v1/upload
# JSON output with boxUrl, thumbnailUrl, manageUrl, deleteUrl, zipUrl, and file entries.
curl -F sharex=@./screenshot.png \
-H 'Accept: application/json' \
http://localhost:8080/api/v1/upload
```
The upload endpoint accepts multipart fields named `file` and `sharex`. ShareX users can start
from `examples/sharex/warpbox-anonymous.sxcu`; update `RequestURL` to match your instance URL.
Authenticated uploads (your account's limits) add an `Authorization: Bearer <token>` header — mint
a token under **Account → Access tokens**. The JSON response uses ShareX placeholders
`{json:boxUrl}` (URL), `{json:thumbnailUrl}` (thumbnail), `{json:deleteUrl}` (deletion), and
`{json:error}` (error message).
### Grouping multiple files into one box (`X-Warpbox-Batch`)
By default every uploaded file becomes its own box. To put several files in a **single** box, send
the opt-in `X-Warpbox-Batch` header: requests that share the same header value (scoped per account,
or per IP for anonymous uploads) within 20s are appended to the same box. This lets a multi-file
ShareX selection — which ShareX sends as separate back-to-back requests — land as one shareable
link. The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file.
Requests without the header behave exactly as before.
## Stage 4 Accounts + Personal Boxes
- `/register` bootstraps the first admin account only when no users exist.
- `/login` and `/logout` provide cookie-based web sessions.
- `/app` is the personal dashboard for logged-in users, showing owned boxes, storage usage, upload
history, and flat collections. Uploading still happens from the homepage.
- `/admin/users` lets admins create invite links, disable/reactivate users, and generate reset links.
- Logged-in browser uploads from `/` still use `POST /api/v1/upload`, but the resulting box is
stored with owner and optional collection metadata.
- Admin users are exempt from the global max upload size on the homepage upload flow. Future
per-user quotas should apply to this same upload path rather than creating a second uploader.
- `/admin/settings` controls anonymous uploads, anonymous max upload size, daily upload caps, default
user storage quota, and usage retention.
- `/admin/users` shows storage/daily usage and lets admins set per-user storage quota overrides.
- `/admin/storage` manages the built-in local file backend and S3-compatible bucket backends.
- `/admin/bans` manages manual IP/CIDR bans and optional automatic bans for suspicious probes and
repeated login failures. Auto-ban is off by default and configured from the admin UI.
- Upload limits now include daily bytes, daily box counts, active box counts, short-window request
limits, max expiration days, local storage capacity in GB, and per-user policy overrides.
- Uploaded file content, thumbnails, and private box metadata use the selected storage backend.
The bbolt database and JSON logs remain local under `./data/db` and `./data/logs`.
- Anonymous uploads, ShareX uploads, unlisted public box links, password protection, expiry, delete
tokens, thumbnails, and cleanup continue to work as before.
Email delivery is intentionally deferred. Invite and reset links are copyable today; future SMTP
support will power public forgot-password and optional email delivery.
## Runtime Data
Warpbox keeps local runtime data under the configured data directory:
- `data/files/{box_id}/@each@{file_id}.ext` - uploaded file contents when the local backend is selected.
- `data/files/{box_id}/@thumb@{file_id}.jpg` - generated previews when the local backend is selected.
- `data/db/warpbox.bbolt` - bbolt metadata database for boxes and file records.
- `data/db/warpbox.bbolt` also stores users, sessions, invites, and collections.
- `data/db/warpbox.bbolt` stores upload policy settings and daily usage records keyed by plain IP
for anonymous uploads and user ID for signed-in uploads.
- `data/db/warpbox.bbolt` stores manual bans, automatic ban settings, abuse counters, and malicious
path rules.
- `data/logs/{YYYY-MM-DD}.log` - JSONL logs, one event per line.
## Static Asset Policy
The static handler sets long-lived immutable caching for images, video, audio, and fonts, shorter caching for CSS/JS, and gzip compression for compressible responses.
The static handler sets long-lived immutable caching for images, video, audio, and fonts, shorter
caching for CSS/JS, and gzip compression for compressible responses.

View File

@@ -11,27 +11,30 @@ import (
)
type Config struct {
AppName string
AppVersion string
Environment string
Addr string
BaseURL string
DataDir string
AdminToken string
StaticDir string
TemplateDir string
ReadHeaderTimeout time.Duration
ReadTimeout time.Duration
WriteTimeout time.Duration
IdleTimeout time.Duration
TrustedProxies []string
JobsEnabled bool
CleanupEnabled bool
CleanupEvery time.Duration
ThumbnailEnabled bool
ThumbnailEvery time.Duration
MaxUploadSize int64
DefaultSettings SettingsDefaults
AppName string
AppVersion string
Environment string
Addr string
BaseURL string
DataDir string
AdminToken string
StaticDir string
TemplateDir string
ReadHeaderTimeout time.Duration
ReadTimeout time.Duration
WriteTimeout time.Duration
IdleTimeout time.Duration
TrustedProxies []string
JobsEnabled bool
CleanupEnabled bool
CleanupEvery time.Duration
ThumbnailEnabled bool
ThumbnailEvery time.Duration
ResumableUploadsEnabled bool
ResumableChunkSize int64
ResumableRetention time.Duration
MaxUploadSize int64
DefaultSettings SettingsDefaults
}
type SettingsDefaults struct {
@@ -52,30 +55,38 @@ type SettingsDefaults struct {
ShortWindowSeconds int
AnonymousStorageBackend string
UserStorageBackend string
ResumableUploadsEnabled bool
ResumableChunkSizeMB float64
ResumableRetentionHours int
ResumableChunkMode string
ResumableChunkPath string
}
func Load() (Config, error) {
cfg := Config{
AppName: envString("WARPBOX_APP_NAME", "warpbox.dev"),
AppVersion: envString("APP_VERSION", "dev"),
Environment: envString("WARPBOX_ENV", "development"),
Addr: envString("WARPBOX_ADDR", ":8080"),
BaseURL: strings.TrimRight(envString("WARPBOX_BASE_URL", "http://localhost:8080"), "/"),
DataDir: envString("WARPBOX_DATA_DIR", defaultPath("data")),
AdminToken: envString("WARPBOX_ADMIN_TOKEN", ""),
StaticDir: envString("WARPBOX_STATIC_DIR", defaultPath("static")),
TemplateDir: envString("WARPBOX_TEMPLATE_DIR", defaultPath("templates")),
ReadHeaderTimeout: envDuration("WARPBOX_READ_HEADER_TIMEOUT", 15*time.Second),
ReadTimeout: envDuration("WARPBOX_READ_TIMEOUT", 0),
WriteTimeout: envDuration("WARPBOX_WRITE_TIMEOUT", 0),
IdleTimeout: envDuration("WARPBOX_IDLE_TIMEOUT", 120*time.Second),
TrustedProxies: envCSV("WARPBOX_TRUSTED_PROXIES"),
JobsEnabled: envBool("WARPBOX_JOBS_ENABLED", true),
CleanupEnabled: envBool("WARPBOX_CLEANUP_ENABLED", true),
CleanupEvery: envDuration("WARPBOX_CLEANUP_EVERY", time.Hour),
ThumbnailEnabled: envBool("WARPBOX_THUMBNAIL_ENABLED", true),
ThumbnailEvery: envDuration("WARPBOX_THUMBNAIL_EVERY", time.Minute),
MaxUploadSize: envMegabytes("WARPBOX_MAX_UPLOAD_SIZE_MB", 2048), // 2 GiB default.
AppName: envString("WARPBOX_APP_NAME", "warpbox.dev"),
AppVersion: envString("APP_VERSION", "dev"),
Environment: envString("WARPBOX_ENV", "development"),
Addr: envString("WARPBOX_ADDR", ":8080"),
BaseURL: strings.TrimRight(envString("WARPBOX_BASE_URL", "http://localhost:8080"), "/"),
DataDir: envString("WARPBOX_DATA_DIR", defaultPath("data")),
AdminToken: envString("WARPBOX_ADMIN_TOKEN", ""),
StaticDir: envString("WARPBOX_STATIC_DIR", defaultPath("static")),
TemplateDir: envString("WARPBOX_TEMPLATE_DIR", defaultPath("templates")),
ReadHeaderTimeout: envDuration("WARPBOX_READ_HEADER_TIMEOUT", 15*time.Second),
ReadTimeout: envDuration("WARPBOX_READ_TIMEOUT", 0),
WriteTimeout: envDuration("WARPBOX_WRITE_TIMEOUT", 0),
IdleTimeout: envDuration("WARPBOX_IDLE_TIMEOUT", 120*time.Second),
TrustedProxies: envCSV("WARPBOX_TRUSTED_PROXIES"),
JobsEnabled: envBool("WARPBOX_JOBS_ENABLED", true),
CleanupEnabled: envBool("WARPBOX_CLEANUP_ENABLED", true),
CleanupEvery: envDuration("WARPBOX_CLEANUP_EVERY", time.Hour),
ThumbnailEnabled: envBool("WARPBOX_THUMBNAIL_ENABLED", true),
ThumbnailEvery: envDuration("WARPBOX_THUMBNAIL_EVERY", time.Minute),
ResumableUploadsEnabled: envBool("WARPBOX_RESUMABLE_UPLOADS_ENABLED", true),
ResumableChunkSize: envMegabytes("WARPBOX_RESUMABLE_CHUNK_MB", 8),
ResumableRetention: time.Duration(envInt("WARPBOX_RESUMABLE_RETENTION_HOURS", 24)) * time.Hour,
MaxUploadSize: envMegabytes("WARPBOX_MAX_UPLOAD_SIZE_MB", 2048), // 2 GiB default.
DefaultSettings: SettingsDefaults{
AnonymousUploadsEnabled: envBool("WARPBOX_ANONYMOUS_UPLOADS_ENABLED", true),
AnonymousMaxUploadMB: envMegabytesLimitFloat("WARPBOX_ANONYMOUS_MAX_UPLOAD_MB", 512),
@@ -94,8 +105,13 @@ func Load() (Config, error) {
ShortWindowSeconds: envInt("WARPBOX_SHORT_WINDOW_SECONDS", 60),
AnonymousStorageBackend: envString("WARPBOX_ANONYMOUS_STORAGE_BACKEND", "local"),
UserStorageBackend: envString("WARPBOX_USER_STORAGE_BACKEND", "local"),
ResumableChunkMode: envString("WARPBOX_RESUMABLE_CHUNK_MODE", "same"),
ResumableChunkPath: envString("WARPBOX_RESUMABLE_CHUNK_PATH", ""),
},
}
cfg.DefaultSettings.ResumableUploadsEnabled = cfg.ResumableUploadsEnabled
cfg.DefaultSettings.ResumableChunkSizeMB = float64(cfg.ResumableChunkSize) / 1024 / 1024
cfg.DefaultSettings.ResumableRetentionHours = int(cfg.ResumableRetention / time.Hour)
if cfg.BaseURL == "" {
return Config{}, fmt.Errorf("WARPBOX_BASE_URL cannot be empty")
@@ -103,6 +119,12 @@ func Load() (Config, error) {
if cfg.MaxUploadSize <= 0 {
return Config{}, fmt.Errorf("WARPBOX_MAX_UPLOAD_SIZE_MB must be positive")
}
if cfg.ResumableChunkSize <= 0 {
return Config{}, fmt.Errorf("WARPBOX_RESUMABLE_CHUNK_MB must be positive")
}
if cfg.ResumableRetention <= 0 {
return Config{}, fmt.Errorf("WARPBOX_RESUMABLE_RETENTION_HOURS must be positive")
}
if !validUnlimitedMegabyteLimit(cfg.DefaultSettings.AnonymousMaxUploadMB) ||
!validUnlimitedMegabyteLimit(cfg.DefaultSettings.AnonymousDailyUploadMB) ||
!validUnlimitedMegabyteLimit(cfg.DefaultSettings.UserDailyUploadMB) ||

View File

@@ -68,4 +68,13 @@ func TestLoadDefaultsUseLargeUploadFriendlyTimeouts(t *testing.T) {
if cfg.WriteTimeout != 0 {
t.Fatalf("WriteTimeout = %s, want 0 for long uploads", cfg.WriteTimeout)
}
if !cfg.ResumableUploadsEnabled {
t.Fatalf("ResumableUploadsEnabled = false, want true")
}
if cfg.ResumableChunkSize != 8*1024*1024 {
t.Fatalf("ResumableChunkSize = %d, want 8 MiB", cfg.ResumableChunkSize)
}
if cfg.ResumableRetention != 24*time.Hour {
t.Fatalf("ResumableRetention = %s, want 24h", cfg.ResumableRetention)
}
}

View File

@@ -574,6 +574,7 @@ func (a *App) AdminSettingsPost(w http.ResponseWriter, r *http.Request) {
return
}
settings.AnonymousUploadsEnabled = r.FormValue("anonymous_uploads_enabled") == "on"
settings.ResumableUploadsEnabled = r.FormValue("resumable_uploads_enabled") == "on"
if value := parsePositiveInt(r.FormValue("usage_retention_days")); value > 0 {
settings.UsageRetentionDays = value
}
@@ -604,6 +605,16 @@ func (a *App) AdminSettingsPost(w http.ResponseWriter, r *http.Request) {
if value := parsePositiveInt(r.FormValue("short_window_seconds")); value > 0 {
settings.ShortWindowSeconds = value
}
if value := parsePositiveFloat(r.FormValue("resumable_chunk_size_mb")); value > 0 {
settings.ResumableChunkSizeMB = value
}
if value := parsePositiveInt(r.FormValue("resumable_retention_hours")); value > 0 {
settings.ResumableRetentionHours = value
}
if value := strings.TrimSpace(r.FormValue("resumable_chunk_mode")); value != "" {
settings.ResumableChunkMode = value
}
settings.ResumableChunkPath = strings.TrimSpace(r.FormValue("resumable_chunk_path"))
if value := r.FormValue("anonymous_storage_backend"); value != "" {
settings.AnonymousStorageBackend = value
}

View File

@@ -141,6 +141,13 @@ func (a *App) RegisterRoutes(mux *http.ServeMux) {
mux.HandleFunc("GET /api/v1/schemas/upload-request.json", a.UploadRequestSchema)
mux.HandleFunc("GET /api/v1/schemas/upload-response.json", a.UploadResponseSchema)
mux.HandleFunc("POST /api/v1/upload", a.Upload)
mux.HandleFunc("POST /api/v1/uploads/resumable", a.CreateResumableUpload)
mux.HandleFunc("GET /api/v1/uploads/resumable/{sessionID}", a.ResumableUploadStatus)
mux.HandleFunc("POST /api/v1/uploads/resumable/{sessionID}/files", a.AddResumableFiles)
mux.HandleFunc("PUT /api/v1/uploads/resumable/{sessionID}/files/{fileID}/chunks/{index}", a.PutResumableChunk)
mux.HandleFunc("POST /api/v1/uploads/resumable/{sessionID}/complete", a.CompleteResumableUpload)
mux.HandleFunc("POST /api/v1/uploads/resumable/{sessionID}/complete-uploaded", a.CompleteUploadedResumableUpload)
mux.HandleFunc("DELETE /api/v1/uploads/resumable/{sessionID}", a.CancelResumableUpload)
mux.HandleFunc("GET /emoji/{pack}/{file}", a.EmojiAsset)
mux.Handle("GET /static/", a.Static())
}

View File

@@ -52,6 +52,7 @@ type fileView struct {
Reactions []reactionView
ReactionMore int
Reacted bool
Processing bool
}
type reactionView struct {
@@ -101,6 +102,15 @@ func (a *App) DownloadPage(w http.ResponseWriter, r *http.Request) {
return
}
locked := a.uploadService.IsProtected(box) && !a.isBoxUnlocked(r, box)
if isSocialPreviewBot(r) && !locked && len(box.Files) == 1 {
if box.Files[0].Processing {
http.Error(w, "file is still processing", http.StatusAccepted)
return
}
a.serveFileContent(w, r, box, box.Files[0], false)
a.logger.Info("single-file box served inline for social preview", withRequestLogAttrs(r, "source", "download", "severity", "user_activity", "code", 2008, "box_id", box.ID, "file_id", box.Files[0].ID)...)
return
}
visitorID := a.reactionVisitorID(w, r)
reactionsByFile, reactedByFile, err := a.reactionService.SummaryForBox(box.ID, visitorID)
if err != nil {
@@ -159,6 +169,15 @@ func (a *App) DownloadFile(w http.ResponseWriter, r *http.Request) {
}
locked := a.uploadService.IsProtected(box) && !a.isBoxUnlocked(r, box)
if isSocialPreviewBot(r) && !locked {
if file.Processing {
http.Error(w, "file is still processing", http.StatusAccepted)
return
}
a.serveFileContent(w, r, box, file, false)
a.logger.Info("file served inline for social preview", withRequestLogAttrs(r, "source", "download", "severity", "user_activity", "code", 2009, "box_id", box.ID, "file_id", file.ID)...)
return
}
view := a.fileView(box, file)
title := file.Name
description := fmt.Sprintf("%s shared via Warpbox", helpers.FormatBytes(file.Size))
@@ -193,6 +212,10 @@ func (a *App) DownloadFileContent(w http.ResponseWriter, r *http.Request) {
http.Error(w, "password required", http.StatusUnauthorized)
return
}
if file.Processing {
http.Error(w, "file is still processing", http.StatusAccepted)
return
}
a.serveFileContent(w, r, box, file, r.URL.Query().Get("inline") != "1")
a.logger.Info("file content served", withRequestLogAttrs(r, "source", "download", "severity", "user_activity", "code", 2005, "box_id", box.ID, "file_id", file.ID, "attachment", r.URL.Query().Get("inline") != "1")...)
@@ -373,6 +396,7 @@ func (a *App) fileViewWithReactions(box services.Box, file services.File, reacti
Reactions: reactionViews,
ReactionMore: reactionOverflowCount(reactionViews),
Reacted: reacted,
Processing: file.Processing,
}
}
@@ -583,3 +607,31 @@ func absoluteURL(r *http.Request, path string) string {
}
return fmt.Sprintf("%s://%s%s", scheme, r.Host, path)
}
func isSocialPreviewBot(r *http.Request) bool {
agent := strings.ToLower(r.UserAgent())
if agent == "" {
return false
}
bots := []string{
"discordbot",
"twitterbot",
"facebookexternalhit",
"telegrambot",
"whatsapp",
"slackbot",
"linkedinbot",
"skypeuripreview",
"embedly",
"pinterest",
"vkshare",
"mattermost",
"mastodon",
}
for _, bot := range bots {
if strings.Contains(agent, bot) {
return true
}
}
return false
}

View File

@@ -0,0 +1,427 @@
package handlers
import (
"context"
"encoding/json"
"fmt"
"net/http"
"strconv"
"strings"
"time"
"warpbox.dev/backend/libs/helpers"
"warpbox.dev/backend/libs/jobs"
"warpbox.dev/backend/libs/services"
)
type resumableCreateRequest struct {
Files []services.ResumableFileInput `json:"files"`
MaxDays int `json:"maxDays"`
ExpiresMinutes int `json:"expiresMinutes"`
MaxDownloads int `json:"maxDownloads"`
Password string `json:"password"`
ObfuscateMetadata bool `json:"obfuscateMetadata"`
CollectionID string `json:"collectionId"`
}
type resumableSessionResponse struct {
SessionID string `json:"sessionId"`
ResumeToken string `json:"resumeToken,omitempty"`
ChunkSize int64 `json:"chunkSize"`
Status string `json:"status"`
BoxID string `json:"boxId,omitempty"`
ExpiresAt string `json:"expiresAt"`
Files []services.ResumableFile `json:"files"`
}
func (a *App) CreateResumableUpload(w http.ResponseWriter, r *http.Request) {
user, loggedIn, authErr := a.currentUserWithAuthError(r)
if authErr != nil {
a.logger.Warn("resumable upload rejected invalid bearer token", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4011)...)
helpers.WriteJSONError(w, http.StatusUnauthorized, "invalid bearer token")
return
}
isAdminUpload := loggedIn && user.Role == services.UserRoleAdmin
settings, policy, ok := a.loadUploadPolicyForAPI(w, r, user, loggedIn)
if !ok {
return
}
if !settings.ResumableUploadsEnabled {
helpers.WriteJSONError(w, http.StatusForbidden, "resumable uploads are disabled")
return
}
if !loggedIn && !settings.AnonymousUploadsEnabled {
a.logger.Warn("resumable anonymous upload rejected disabled", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4013)...)
helpers.WriteJSONError(w, http.StatusForbidden, "anonymous uploads are disabled")
return
}
rateKey := uploadRateKey(r, user, loggedIn)
if !isAdminUpload && policy.ShortRequests > 0 && !a.rateLimiter.Allow("upload:"+rateKey, policy.ShortRequests, policy.ShortWindow, time.Now().UTC()) {
a.logger.Warn("resumable upload rate limited", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4291, "user_id", user.ID)...)
helpers.WriteJSONError(w, http.StatusTooManyRequests, "too many upload requests, please slow down")
return
}
var payload resumableCreateRequest
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
helpers.WriteJSONError(w, http.StatusBadRequest, "upload session request could not be read")
return
}
fileSizes := make([]int64, 0, len(payload.Files))
var totalBytes int64
for _, file := range payload.Files {
fileSizes = append(fileSizes, file.Size)
totalBytes += file.Size
}
if !isAdminUpload {
if status, message := a.checkUploadPolicyForSizes(r, user, loggedIn, settings, policy, fileSizes, totalBytes); message != "" {
a.logger.Warn("resumable upload rejected by policy", withRequestLogAttrs(r, "source", "quota", "severity", "warn", "code", status, "user_id", user.ID, "message", message, "bytes", totalBytes, "files", len(payload.Files))...)
helpers.WriteJSONError(w, status, message)
return
}
if status, message := a.checkBoxCreationPolicy(r, user, loggedIn, policy); message != "" {
a.logger.Warn("resumable upload rejected by box policy", withRequestLogAttrs(r, "source", "quota", "severity", "warn", "code", status, "user_id", user.ID, "message", message, "bytes", totalBytes, "files", len(payload.Files))...)
helpers.WriteJSONError(w, status, message)
return
}
}
opts, err := a.resumableUploadOptions(r, payload, user, loggedIn, isAdminUpload, policy)
if err != nil {
helpers.WriteJSONError(w, http.StatusRequestEntityTooLarge, err.Error())
return
}
chunkSize := int64(settings.ResumableChunkSizeMB * 1024 * 1024)
retention := time.Duration(settings.ResumableRetentionHours) * time.Hour
session, err := a.uploadService.CreateResumableSession(payload.Files, opts, chunkSize, retention, resumableChunkRoot(settings))
if err != nil {
a.logger.Warn("resumable session create failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4002, "user_id", user.ID, "error", err.Error())...)
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
return
}
a.logger.Info("resumable upload session created", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2002, "user_id", user.ID, "session_id", session.ID, "files", len(session.Files), "bytes", totalBytes, "anonymous", !loggedIn)...)
helpers.WriteJSON(w, http.StatusCreated, resumableResponse(session))
}
func (a *App) ResumableUploadStatus(w http.ResponseWriter, r *http.Request) {
session, ok := a.authorizedResumableSession(w, r)
if !ok {
return
}
helpers.WriteJSON(w, http.StatusOK, resumableResponse(session))
}
func (a *App) AddResumableFiles(w http.ResponseWriter, r *http.Request) {
session, ok := a.authorizedResumableSession(w, r)
if !ok {
return
}
user, loggedIn, _ := a.currentUserWithAuthError(r)
isAdminUpload := loggedIn && user.Role == services.UserRoleAdmin
settings, policy, ok := a.loadUploadPolicyForAPI(w, r, user, loggedIn)
if !ok {
return
}
var payload struct {
Files []services.ResumableFileInput `json:"files"`
}
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
helpers.WriteJSONError(w, http.StatusBadRequest, "upload files request could not be read")
return
}
fileSizes := make([]int64, 0, len(session.Files)+len(payload.Files))
var totalBytes int64
for _, file := range session.Files {
fileSizes = append(fileSizes, file.Size)
totalBytes += file.Size
}
for _, file := range payload.Files {
fileSizes = append(fileSizes, file.Size)
totalBytes += file.Size
}
if !isAdminUpload {
if status, message := a.checkUploadPolicyForSizes(r, user, loggedIn, settings, policy, fileSizes, totalBytes); message != "" {
helpers.WriteJSONError(w, status, message)
return
}
}
updated, err := a.uploadService.AddResumableFiles(session.ID, payload.Files)
if err != nil {
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
return
}
a.logger.Info("resumable upload files added", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2006, "session_id", session.ID, "added", len(updated.Files)-len(session.Files), "files", len(updated.Files))...)
helpers.WriteJSON(w, http.StatusOK, resumableResponse(updated))
}
func (a *App) PutResumableChunk(w http.ResponseWriter, r *http.Request) {
session, ok := a.authorizedResumableSession(w, r)
if !ok {
return
}
fileID := r.PathValue("fileID")
index, err := strconv.Atoi(r.PathValue("index"))
if err != nil {
helpers.WriteJSONError(w, http.StatusBadRequest, "chunk index is invalid")
return
}
updated, err := a.uploadService.PutResumableChunk(r.Context(), session.ID, fileID, index, r.Body)
if err != nil {
a.logger.Warn("resumable chunk failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4003, "session_id", session.ID, "file_id", fileID, "chunk", index, "error", err.Error())...)
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
return
}
a.logger.Info("resumable chunk uploaded", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2003, "session_id", session.ID, "file_id", fileID, "chunk", index)...)
helpers.WriteJSON(w, http.StatusOK, resumableResponse(updated))
}
func (a *App) CompleteResumableUpload(w http.ResponseWriter, r *http.Request) {
session, ok := a.authorizedResumableSession(w, r)
if !ok {
return
}
if session.Status == services.ResumableStatusCompleted || session.Status == services.ResumableStatusProcessing {
result, completed, err := a.uploadService.CompleteResumableSession(r.Context(), session.ID)
if err != nil {
a.logger.Warn("resumable upload completion replay failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4004, "session_id", session.ID, "error", err.Error())...)
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
return
}
a.logger.Info("resumable upload completion replayed", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2004, "session_id", completed.ID, "box_id", result.BoxID, "files", len(result.Files))...)
helpers.WriteJSON(w, http.StatusOK, result)
return
}
user, loggedIn, _ := a.currentUserWithAuthError(r)
isAdminUpload := loggedIn && user.Role == services.UserRoleAdmin
settings, policy, ok := a.loadUploadPolicyForAPI(w, r, user, loggedIn)
if !ok {
return
}
fileSizes := make([]int64, 0, len(session.Files))
var totalBytes int64
for _, file := range session.Files {
fileSizes = append(fileSizes, file.Size)
totalBytes += file.Size
}
if !isAdminUpload {
if status, message := a.checkUploadPolicyForSizes(r, user, loggedIn, settings, policy, fileSizes, totalBytes); message != "" {
helpers.WriteJSONError(w, status, message)
return
}
if status, message := a.checkBoxCreationPolicy(r, user, loggedIn, policy); message != "" {
helpers.WriteJSONError(w, status, message)
return
}
if status, message := a.checkStorageBackendCapacity(session.Options.StorageBackendID, settings, totalBytes); message != "" {
helpers.WriteJSONError(w, status, message)
return
}
}
result, completed, err := a.uploadService.CreateProcessingBoxFromResumable(session.ID)
if err != nil {
a.logger.Warn("resumable upload complete failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4004, "session_id", session.ID, "error", err.Error())...)
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
return
}
if !isAdminUpload {
if err := a.recordUploadUsage(r, user, loggedIn, totalBytes, 1); err != nil {
a.logger.Warn("failed to record resumable upload usage", "source", "quota", "severity", "warn", "code", 4404, "error", err.Error())
}
if err := a.settingsService.CleanupUsage(time.Now().UTC(), settings.UsageRetentionDays); err != nil {
a.logger.Warn("failed to cleanup upload usage", "source", "quota", "severity", "warn", "code", 4405, "error", err.Error())
}
}
a.finalizeResumableUploadAsync(completed.ID, result.BoxID)
a.logger.Info("resumable upload queued for processing", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2004, "user_id", user.ID, "session_id", completed.ID, "box_id", result.BoxID, "files", len(result.Files), "bytes", totalBytes, "admin", isAdminUpload, "anonymous", !loggedIn)...)
helpers.WriteJSON(w, http.StatusCreated, result)
}
func (a *App) CompleteUploadedResumableUpload(w http.ResponseWriter, r *http.Request) {
session, ok := a.authorizedResumableSession(w, r)
if !ok {
return
}
user, loggedIn, _ := a.currentUserWithAuthError(r)
isAdminUpload := loggedIn && user.Role == services.UserRoleAdmin
settings, policy, ok := a.loadUploadPolicyForAPI(w, r, user, loggedIn)
if !ok {
return
}
fileSizes := make([]int64, 0, len(session.Files))
var totalBytes int64
var completeCount int
for _, file := range session.Files {
if len(file.UploadedChunks) != file.ChunkCount {
continue
}
fileSizes = append(fileSizes, file.Size)
totalBytes += file.Size
completeCount++
}
if completeCount == 0 {
helpers.WriteJSONError(w, http.StatusBadRequest, "no fully uploaded files to finish")
return
}
if !isAdminUpload {
if status, message := a.checkUploadPolicyForSizes(r, user, loggedIn, settings, policy, fileSizes, totalBytes); message != "" {
helpers.WriteJSONError(w, status, message)
return
}
if status, message := a.checkBoxCreationPolicy(r, user, loggedIn, policy); message != "" {
helpers.WriteJSONError(w, status, message)
return
}
if status, message := a.checkStorageBackendCapacity(session.Options.StorageBackendID, settings, totalBytes); message != "" {
helpers.WriteJSONError(w, status, message)
return
}
}
result, completed, err := a.uploadService.CompleteUploadedResumableSession(r.Context(), session.ID)
if err != nil {
a.logger.Warn("resumable partial complete failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4005, "session_id", session.ID, "error", err.Error())...)
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
return
}
if !isAdminUpload {
if err := a.recordUploadUsage(r, user, loggedIn, totalBytes, 1); err != nil {
a.logger.Warn("failed to record partial resumable upload usage", "source", "quota", "severity", "warn", "code", 4406, "error", err.Error())
}
if err := a.settingsService.CleanupUsage(time.Now().UTC(), settings.UsageRetentionDays); err != nil {
a.logger.Warn("failed to cleanup upload usage", "source", "quota", "severity", "warn", "code", 4405, "error", err.Error())
}
}
jobs.GenerateThumbnailsForBoxAsync(a.uploadService, a.logger, result.BoxID)
a.logger.Info("resumable uploaded files completed", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2007, "user_id", user.ID, "session_id", completed.ID, "box_id", result.BoxID, "files", len(result.Files), "bytes", totalBytes, "admin", isAdminUpload, "anonymous", !loggedIn)...)
helpers.WriteJSON(w, http.StatusCreated, result)
}
func (a *App) finalizeResumableUploadAsync(sessionID, boxID string) {
go func() {
a.logger.Info("resumable upload processing started", "source", "user-upload", "severity", "user_activity", "code", 2009, "session_id", sessionID, "box_id", boxID)
result, err := a.uploadService.FinalizeProcessingResumableSession(context.Background(), sessionID)
if err != nil {
a.logger.Warn("resumable upload processing failed", "source", "user-upload", "severity", "warn", "code", 4010, "session_id", sessionID, "box_id", boxID, "error", err.Error())
return
}
jobs.GenerateThumbnailsForBoxAsync(a.uploadService, a.logger, result.BoxID)
a.logger.Info("resumable upload processing completed", "source", "user-upload", "severity", "user_activity", "code", 2010, "session_id", sessionID, "box_id", result.BoxID, "files", len(result.Files))
}()
}
func resumableChunkRoot(settings services.UploadPolicySettings) string {
if settings.ResumableChunkMode != "custom" {
return ""
}
return strings.TrimSpace(settings.ResumableChunkPath)
}
func (a *App) CancelResumableUpload(w http.ResponseWriter, r *http.Request) {
session, ok := a.authorizedResumableSession(w, r)
if !ok {
return
}
if err := a.uploadService.CancelResumableSession(session.ID); err != nil {
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
return
}
a.logger.Info("resumable upload cancelled", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2005, "session_id", session.ID)...)
w.WriteHeader(http.StatusNoContent)
}
func (a *App) authorizedResumableSession(w http.ResponseWriter, r *http.Request) (services.ResumableSession, bool) {
user, loggedIn, authErr := a.currentUserWithAuthError(r)
if authErr != nil {
helpers.WriteJSONError(w, http.StatusUnauthorized, "invalid bearer token")
return services.ResumableSession{}, false
}
session, err := a.uploadService.GetResumableSession(r.PathValue("sessionID"))
if err != nil {
helpers.WriteJSONError(w, http.StatusNotFound, "upload session not found")
return services.ResumableSession{}, false
}
if !a.uploadService.VerifyResumableToken(session, r.Header.Get("X-Warpbox-Resume-Token")) {
helpers.WriteJSONError(w, http.StatusUnauthorized, "upload session not found")
return services.ResumableSession{}, false
}
if loggedIn {
if session.Options.OwnerID != user.ID {
helpers.WriteJSONError(w, http.StatusForbidden, "upload session not found")
return services.ResumableSession{}, false
}
return session, true
}
if session.Options.OwnerID != "" || session.Options.CreatorIP != uploadClientIP(r) {
helpers.WriteJSONError(w, http.StatusForbidden, "upload session not found")
return services.ResumableSession{}, false
}
return session, true
}
func (a *App) loadUploadPolicyForAPI(w http.ResponseWriter, r *http.Request, user services.User, loggedIn bool) (services.UploadPolicySettings, services.EffectiveUploadPolicy, bool) {
settings, err := a.settingsService.UploadPolicy()
if err != nil {
a.logger.Error("failed to load upload policy", "source", "settings", "severity", "error", "code", 5006, "error", err.Error())
helpers.WriteJSONError(w, http.StatusInternalServerError, "upload policy could not be loaded")
return services.UploadPolicySettings{}, services.EffectiveUploadPolicy{}, false
}
return settings, a.effectiveUploadPolicy(settings, user, loggedIn), true
}
func (a *App) resumableUploadOptions(r *http.Request, payload resumableCreateRequest, user services.User, loggedIn, isAdminUpload bool, policy services.EffectiveUploadPolicy) (services.UploadOptions, error) {
var ownerID string
var collectionID string
if loggedIn {
ownerID = user.ID
collectionID = strings.TrimSpace(payload.CollectionID)
if !a.authService.CollectionOwnedBy(collectionID, user.ID) {
return services.UploadOptions{}, fmt.Errorf("collection not found")
}
}
unlimitedExpiry := isAdminUpload || policy.MaxDays < 0
rawMaxDays := payload.MaxDays
maxDays := rawMaxDays
if maxDays <= 0 {
maxDays = 7
if policy.MaxDays > 0 && policy.MaxDays < maxDays {
maxDays = policy.MaxDays
}
}
expiresMinutes := payload.ExpiresMinutes
if expiresMinutes < 0 || rawMaxDays < 0 {
if !unlimitedExpiry {
return services.UploadOptions{}, fmt.Errorf("expiration cannot exceed %d days", policy.MaxDays)
}
expiresMinutes = -1
} else if !unlimitedExpiry {
if maxDays > policy.MaxDays {
return services.UploadOptions{}, fmt.Errorf("expiration cannot exceed %d days", policy.MaxDays)
}
if expiresMinutes > 0 && expiresMinutes > policy.MaxDays*24*60 {
return services.UploadOptions{}, fmt.Errorf("expiration cannot exceed %d days", policy.MaxDays)
}
}
return services.UploadOptions{
MaxDays: maxDays,
ExpiresInMinutes: expiresMinutes,
MaxDownloads: payload.MaxDownloads,
Password: payload.Password,
ObfuscateMetadata: payload.ObfuscateMetadata,
OwnerID: ownerID,
CollectionID: collectionID,
SkipSizeLimit: isAdminUpload || policy.MaxUploadMB < 0,
CreatorIP: uploadClientIP(r),
StorageBackendID: policy.StorageBackendID,
}, nil
}
func resumableResponse(session services.ResumableSession) resumableSessionResponse {
return resumableSessionResponse{
SessionID: session.ID,
ResumeToken: session.ResumeToken,
ChunkSize: session.ChunkSize,
Status: session.Status,
BoxID: session.BoxID,
ExpiresAt: session.ExpiresAt.Format(time.RFC3339),
Files: session.Files,
}
}

View File

@@ -228,11 +228,22 @@ func (a *App) checkUploadPolicy(r *http.Request, user services.User, loggedIn bo
if len(files) == 0 {
return 0, ""
}
sizes := make([]int64, 0, len(files))
for _, file := range files {
sizes = append(sizes, file.Size)
}
return a.checkUploadPolicyForSizes(r, user, loggedIn, settings, policy, sizes, totalBytes)
}
func (a *App) checkUploadPolicyForSizes(r *http.Request, user services.User, loggedIn bool, settings services.UploadPolicySettings, policy services.EffectiveUploadPolicy, fileSizes []int64, totalBytes int64) (int, string) {
if len(fileSizes) == 0 {
return 0, ""
}
now := time.Now().UTC()
if policy.MaxUploadMB > 0 {
maxBytes := services.MegabytesToBytes(policy.MaxUploadMB)
for _, file := range files {
if file.Size > maxBytes {
for _, fileSize := range fileSizes {
if fileSize > maxBytes {
return http.StatusRequestEntityTooLarge, "file exceeds upload size limit"
}
}

View File

@@ -2,6 +2,7 @@ package handlers
import (
"bytes"
"context"
"encoding/json"
"io"
"log/slog"
@@ -10,8 +11,10 @@ import (
"net/http/httptest"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
"warpbox.dev/backend/libs/config"
"warpbox.dev/backend/libs/services"
@@ -103,6 +106,385 @@ func TestUploadTextResponseReturnsOnlyBoxURL(t *testing.T) {
}
}
func TestSocialPreviewBotGetsRawSingleFileBox(t *testing.T) {
app, cleanup := newTestApp(t)
defer cleanup()
payload := uploadThroughApp(t, app)
request := httptest.NewRequest(http.MethodGet, "/d/"+payload.BoxID, nil)
request.Header.Set("User-Agent", "Discordbot/2.0")
request.SetPathValue("boxID", payload.BoxID)
response := httptest.NewRecorder()
app.DownloadPage(response, request)
if response.Code != http.StatusOK {
t.Fatalf("status = %d, body = %s", response.Code, response.Body.String())
}
if strings.Contains(response.Body.String(), "Shared files on Warpbox") {
t.Fatalf("social preview bot received HTML download page")
}
if response.Body.String() != "hello" {
t.Fatalf("social preview body = %q", response.Body.String())
}
}
func TestSocialPreviewBotGetsRawFilePreview(t *testing.T) {
app, cleanup := newTestApp(t)
defer cleanup()
payload := uploadThroughApp(t, app)
request := httptest.NewRequest(http.MethodGet, "/d/"+payload.BoxID+"/f/"+payload.Files[0].ID, nil)
request.Header.Set("User-Agent", "TelegramBot")
request.SetPathValue("boxID", payload.BoxID)
request.SetPathValue("fileID", payload.Files[0].ID)
response := httptest.NewRecorder()
app.DownloadFile(response, request)
if response.Code != http.StatusOK {
t.Fatalf("status = %d, body = %s", response.Code, response.Body.String())
}
if strings.Contains(response.Body.String(), "preview-title") {
t.Fatalf("social preview bot received HTML preview page")
}
if response.Body.String() != "hello" {
t.Fatalf("social preview body = %q", response.Body.String())
}
}
func TestResumableUploadFlowCreatesNormalBox(t *testing.T) {
app, cleanup := newTestApp(t)
defer cleanup()
createBody := `{"files":[{"name":"note.txt","size":11,"contentType":"text/plain"}],"expiresMinutes":60}`
createRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable", strings.NewReader(createBody))
createRequest.Header.Set("Accept", "application/json")
createResponse := httptest.NewRecorder()
app.CreateResumableUpload(createResponse, createRequest)
if createResponse.Code != http.StatusCreated {
t.Fatalf("create status = %d, body = %s", createResponse.Code, createResponse.Body.String())
}
var session struct {
SessionID string `json:"sessionId"`
ResumeToken string `json:"resumeToken"`
ChunkSize int64 `json:"chunkSize"`
Files []struct {
ID string `json:"id"`
ChunkCount int `json:"chunkCount"`
UploadedChunks []int `json:"uploadedChunks"`
} `json:"files"`
}
if err := json.Unmarshal(createResponse.Body.Bytes(), &session); err != nil {
t.Fatalf("json.Unmarshal session returned error: %v", err)
}
if session.SessionID == "" || session.ResumeToken == "" || session.ChunkSize != 4 || len(session.Files) != 1 || session.Files[0].ChunkCount != 3 {
t.Fatalf("unexpected session response: %+v", session)
}
chunks := map[int]string{1: "o wo", 0: "hell", 2: "rld"}
for index, body := range chunks {
request := httptest.NewRequest(http.MethodPut, "/api/v1/uploads/resumable/"+session.SessionID+"/files/"+session.Files[0].ID+"/chunks/"+strconv.Itoa(index), strings.NewReader(body))
request.SetPathValue("sessionID", session.SessionID)
request.SetPathValue("fileID", session.Files[0].ID)
request.SetPathValue("index", strconv.Itoa(index))
request.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
response := httptest.NewRecorder()
app.PutResumableChunk(response, request)
if response.Code != http.StatusOK {
t.Fatalf("chunk %d status = %d, body = %s", index, response.Code, response.Body.String())
}
}
completeRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
completeRequest.SetPathValue("sessionID", session.SessionID)
completeRequest.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
completeResponse := httptest.NewRecorder()
app.CompleteResumableUpload(completeResponse, completeRequest)
if completeResponse.Code != http.StatusCreated {
t.Fatalf("complete status = %d, body = %s", completeResponse.Code, completeResponse.Body.String())
}
var payload services.UploadResult
if err := json.Unmarshal(completeResponse.Body.Bytes(), &payload); err != nil {
t.Fatalf("json.Unmarshal result returned error: %v", err)
}
replayRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
replayRequest.SetPathValue("sessionID", session.SessionID)
replayRequest.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
replayResponse := httptest.NewRecorder()
app.CompleteResumableUpload(replayResponse, replayRequest)
if replayResponse.Code != http.StatusOK {
t.Fatalf("complete replay status = %d, body = %s", replayResponse.Code, replayResponse.Body.String())
}
var replayPayload services.UploadResult
if err := json.Unmarshal(replayResponse.Body.Bytes(), &replayPayload); err != nil {
t.Fatalf("json.Unmarshal replay result returned error: %v", err)
}
if replayPayload.BoxID != payload.BoxID || replayPayload.BoxURL == "" {
t.Fatalf("unexpected replay result: %+v, original: %+v", replayPayload, payload)
}
box := waitForProcessedBox(t, app, payload.BoxID)
if len(box.Files) != 1 || box.Files[0].Name != "note.txt" || box.Files[0].Size != 11 {
t.Fatalf("unexpected box files: %+v", box.Files)
}
object, err := app.uploadService.OpenFileObject(context.Background(), box, box.Files[0])
if err != nil {
t.Fatalf("OpenFileObject returned error: %v", err)
}
data, err := io.ReadAll(object.Body)
object.Body.Close()
if err != nil {
t.Fatalf("ReadAll returned error: %v", err)
}
if string(data) != "hello world" {
t.Fatalf("uploaded body = %q", string(data))
}
}
func TestResumableUploadRequiresAllChunks(t *testing.T) {
app, cleanup := newTestApp(t)
defer cleanup()
createRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable", strings.NewReader(`{"files":[{"name":"note.txt","size":8,"contentType":"text/plain"}]}`))
createResponse := httptest.NewRecorder()
app.CreateResumableUpload(createResponse, createRequest)
if createResponse.Code != http.StatusCreated {
t.Fatalf("create status = %d, body = %s", createResponse.Code, createResponse.Body.String())
}
var session struct {
SessionID string `json:"sessionId"`
ResumeToken string `json:"resumeToken"`
Files []struct {
ID string `json:"id"`
} `json:"files"`
}
if err := json.Unmarshal(createResponse.Body.Bytes(), &session); err != nil {
t.Fatalf("json.Unmarshal session returned error: %v", err)
}
chunkRequest := httptest.NewRequest(http.MethodPut, "/api/v1/uploads/resumable/"+session.SessionID+"/files/"+session.Files[0].ID+"/chunks/0", strings.NewReader("hell"))
chunkRequest.SetPathValue("sessionID", session.SessionID)
chunkRequest.SetPathValue("fileID", session.Files[0].ID)
chunkRequest.SetPathValue("index", "0")
chunkRequest.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
chunkResponse := httptest.NewRecorder()
app.PutResumableChunk(chunkResponse, chunkRequest)
if chunkResponse.Code != http.StatusOK {
t.Fatalf("chunk status = %d, body = %s", chunkResponse.Code, chunkResponse.Body.String())
}
completeRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
completeRequest.SetPathValue("sessionID", session.SessionID)
completeRequest.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
completeResponse := httptest.NewRecorder()
app.CompleteResumableUpload(completeResponse, completeRequest)
if completeResponse.Code != http.StatusBadRequest {
t.Fatalf("complete status = %d, body = %s", completeResponse.Code, completeResponse.Body.String())
}
}
func TestResumableStatusRequiresResumeToken(t *testing.T) {
app, cleanup := newTestApp(t)
defer cleanup()
createRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable", strings.NewReader(`{"files":[{"name":"note.txt","size":4,"contentType":"text/plain"}]}`))
createResponse := httptest.NewRecorder()
app.CreateResumableUpload(createResponse, createRequest)
if createResponse.Code != http.StatusCreated {
t.Fatalf("create status = %d, body = %s", createResponse.Code, createResponse.Body.String())
}
var session struct {
SessionID string `json:"sessionId"`
ResumeToken string `json:"resumeToken"`
}
if err := json.Unmarshal(createResponse.Body.Bytes(), &session); err != nil {
t.Fatalf("json.Unmarshal session returned error: %v", err)
}
missing := httptest.NewRequest(http.MethodGet, "/api/v1/uploads/resumable/"+session.SessionID, nil)
missing.SetPathValue("sessionID", session.SessionID)
missingResponse := httptest.NewRecorder()
app.ResumableUploadStatus(missingResponse, missing)
if missingResponse.Code != http.StatusUnauthorized {
t.Fatalf("missing token status = %d, body = %s", missingResponse.Code, missingResponse.Body.String())
}
wrong := httptest.NewRequest(http.MethodGet, "/api/v1/uploads/resumable/"+session.SessionID, nil)
wrong.SetPathValue("sessionID", session.SessionID)
wrong.Header.Set("X-Warpbox-Resume-Token", "wrong")
wrongResponse := httptest.NewRecorder()
app.ResumableUploadStatus(wrongResponse, wrong)
if wrongResponse.Code != http.StatusUnauthorized {
t.Fatalf("wrong token status = %d, body = %s", wrongResponse.Code, wrongResponse.Body.String())
}
valid := httptest.NewRequest(http.MethodGet, "/api/v1/uploads/resumable/"+session.SessionID, nil)
valid.SetPathValue("sessionID", session.SessionID)
valid.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
validResponse := httptest.NewRecorder()
app.ResumableUploadStatus(validResponse, valid)
if validResponse.Code != http.StatusOK {
t.Fatalf("valid token status = %d, body = %s", validResponse.Code, validResponse.Body.String())
}
if strings.Contains(validResponse.Body.String(), "resumeTokenHash") {
t.Fatalf("status response leaked token hash: %s", validResponse.Body.String())
}
}
func TestResumableUploadCanAddFilesToExistingSession(t *testing.T) {
app, cleanup := newTestApp(t)
defer cleanup()
createBody := `{"files":[{"name":"one.txt","size":4,"contentType":"text/plain","fingerprint":"one"}],"expiresMinutes":60}`
createRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable", strings.NewReader(createBody))
createResponse := httptest.NewRecorder()
app.CreateResumableUpload(createResponse, createRequest)
if createResponse.Code != http.StatusCreated {
t.Fatalf("create status = %d, body = %s", createResponse.Code, createResponse.Body.String())
}
var session struct {
SessionID string `json:"sessionId"`
ResumeToken string `json:"resumeToken"`
Files []struct {
ID string `json:"id"`
} `json:"files"`
}
if err := json.Unmarshal(createResponse.Body.Bytes(), &session); err != nil {
t.Fatalf("json.Unmarshal session returned error: %v", err)
}
firstChunk := httptest.NewRequest(http.MethodPut, "/api/v1/uploads/resumable/"+session.SessionID+"/files/"+session.Files[0].ID+"/chunks/0", strings.NewReader("one!"))
firstChunk.SetPathValue("sessionID", session.SessionID)
firstChunk.SetPathValue("fileID", session.Files[0].ID)
firstChunk.SetPathValue("index", "0")
firstChunk.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
firstChunkResponse := httptest.NewRecorder()
app.PutResumableChunk(firstChunkResponse, firstChunk)
if firstChunkResponse.Code != http.StatusOK {
t.Fatalf("first chunk status = %d, body = %s", firstChunkResponse.Code, firstChunkResponse.Body.String())
}
addRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/files", strings.NewReader(`{"files":[{"name":"two.txt","size":4,"contentType":"text/plain","fingerprint":"two"}]}`))
addRequest.SetPathValue("sessionID", session.SessionID)
addRequest.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
addResponse := httptest.NewRecorder()
app.AddResumableFiles(addResponse, addRequest)
if addResponse.Code != http.StatusOK {
t.Fatalf("add status = %d, body = %s", addResponse.Code, addResponse.Body.String())
}
var updated struct {
Files []struct {
ID string `json:"id"`
Name string `json:"name"`
UploadedChunks []int `json:"uploadedChunks"`
} `json:"files"`
}
if err := json.Unmarshal(addResponse.Body.Bytes(), &updated); err != nil {
t.Fatalf("json.Unmarshal updated returned error: %v", err)
}
if len(updated.Files) != 2 || len(updated.Files[0].UploadedChunks) != 1 {
t.Fatalf("unexpected updated session: %+v", updated)
}
secondChunk := httptest.NewRequest(http.MethodPut, "/api/v1/uploads/resumable/"+session.SessionID+"/files/"+updated.Files[1].ID+"/chunks/0", strings.NewReader("two!"))
secondChunk.SetPathValue("sessionID", session.SessionID)
secondChunk.SetPathValue("fileID", updated.Files[1].ID)
secondChunk.SetPathValue("index", "0")
secondChunk.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
secondChunkResponse := httptest.NewRecorder()
app.PutResumableChunk(secondChunkResponse, secondChunk)
if secondChunkResponse.Code != http.StatusOK {
t.Fatalf("second chunk status = %d, body = %s", secondChunkResponse.Code, secondChunkResponse.Body.String())
}
completeRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
completeRequest.SetPathValue("sessionID", session.SessionID)
completeRequest.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
completeResponse := httptest.NewRecorder()
app.CompleteResumableUpload(completeResponse, completeRequest)
if completeResponse.Code != http.StatusCreated {
t.Fatalf("complete status = %d, body = %s", completeResponse.Code, completeResponse.Body.String())
}
var payload services.UploadResult
if err := json.Unmarshal(completeResponse.Body.Bytes(), &payload); err != nil {
t.Fatalf("json.Unmarshal result returned error: %v", err)
}
box, err := app.uploadService.GetBox(payload.BoxID)
if err != nil {
t.Fatalf("GetBox returned error: %v", err)
}
if len(box.Files) != 2 {
t.Fatalf("box file count = %d, want 2", len(box.Files))
}
}
func TestResumableCompleteUploadedRequiresTokenAndKeepsFinishedFiles(t *testing.T) {
app, cleanup := newTestApp(t)
defer cleanup()
createBody := `{"files":[{"name":"done.txt","size":4,"contentType":"text/plain","fingerprint":"done"},{"name":"partial.txt","size":8,"contentType":"text/plain","fingerprint":"partial"}],"expiresMinutes":60}`
createRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable", strings.NewReader(createBody))
createResponse := httptest.NewRecorder()
app.CreateResumableUpload(createResponse, createRequest)
if createResponse.Code != http.StatusCreated {
t.Fatalf("create status = %d, body = %s", createResponse.Code, createResponse.Body.String())
}
var session struct {
SessionID string `json:"sessionId"`
ResumeToken string `json:"resumeToken"`
Files []struct {
ID string `json:"id"`
} `json:"files"`
}
if err := json.Unmarshal(createResponse.Body.Bytes(), &session); err != nil {
t.Fatalf("json.Unmarshal session returned error: %v", err)
}
for _, chunk := range []struct {
fileIndex int
index string
body string
}{
{fileIndex: 0, index: "0", body: "done"},
{fileIndex: 1, index: "0", body: "part"},
} {
request := httptest.NewRequest(http.MethodPut, "/api/v1/uploads/resumable/"+session.SessionID+"/files/"+session.Files[chunk.fileIndex].ID+"/chunks/"+chunk.index, strings.NewReader(chunk.body))
request.SetPathValue("sessionID", session.SessionID)
request.SetPathValue("fileID", session.Files[chunk.fileIndex].ID)
request.SetPathValue("index", chunk.index)
request.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
response := httptest.NewRecorder()
app.PutResumableChunk(response, request)
if response.Code != http.StatusOK {
t.Fatalf("chunk status = %d, body = %s", response.Code, response.Body.String())
}
}
missing := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete-uploaded", nil)
missing.SetPathValue("sessionID", session.SessionID)
missingResponse := httptest.NewRecorder()
app.CompleteUploadedResumableUpload(missingResponse, missing)
if missingResponse.Code != http.StatusUnauthorized {
t.Fatalf("missing token status = %d, body = %s", missingResponse.Code, missingResponse.Body.String())
}
complete := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete-uploaded", nil)
complete.SetPathValue("sessionID", session.SessionID)
complete.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
completeResponse := httptest.NewRecorder()
app.CompleteUploadedResumableUpload(completeResponse, complete)
if completeResponse.Code != http.StatusCreated {
t.Fatalf("complete-uploaded status = %d, body = %s", completeResponse.Code, completeResponse.Body.String())
}
var payload services.UploadResult
if err := json.Unmarshal(completeResponse.Body.Bytes(), &payload); err != nil {
t.Fatalf("json.Unmarshal result returned error: %v", err)
}
box, err := app.uploadService.GetBox(payload.BoxID)
if err != nil {
t.Fatalf("GetBox returned error: %v", err)
}
if len(box.Files) != 1 || box.Files[0].Name != "done.txt" {
t.Fatalf("complete-uploaded box files = %+v", box.Files)
}
if _, err := app.uploadService.GetResumableSession(session.SessionID); !os.IsNotExist(err) {
t.Fatalf("GetResumableSession after complete-uploaded error = %v, want os.ErrNotExist", err)
}
}
func TestManageBoxAndDeleteFlow(t *testing.T) {
app, cleanup := newTestApp(t)
defer cleanup()
@@ -214,13 +596,16 @@ func newTestApp(t *testing.T) (*App, func()) {
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
cfg := config.Config{
AppName: "warpbox.dev",
AppVersion: "test",
BaseURL: "http://example.test",
DataDir: filepath.Join(root, "data"),
StaticDir: staticDir,
TemplateDir: templateDir,
MaxUploadSize: 1024 * 1024,
AppName: "warpbox.dev",
AppVersion: "test",
BaseURL: "http://example.test",
DataDir: filepath.Join(root, "data"),
StaticDir: staticDir,
TemplateDir: templateDir,
MaxUploadSize: 1024 * 1024,
ResumableUploadsEnabled: true,
ResumableChunkSize: 4,
ResumableRetention: time.Hour,
DefaultSettings: config.SettingsDefaults{
AnonymousUploadsEnabled: true,
AnonymousMaxUploadMB: 1,
@@ -228,6 +613,10 @@ func newTestApp(t *testing.T) (*App, func()) {
UserDailyUploadMB: 8,
DefaultUserStorageMB: 16,
UsageRetentionDays: 30,
ResumableUploadsEnabled: true,
ResumableChunkSizeMB: 0.000003814697265625,
ResumableRetentionHours: 1,
ResumableChunkMode: "same",
},
}
service, err := services.NewUploadService(cfg.MaxUploadSize, cfg.DataDir, cfg.BaseURL, logger)
@@ -342,6 +731,31 @@ func tokenFromURL(t *testing.T, value string) string {
return parts[len(parts)-1]
}
func waitForProcessedBox(t *testing.T, app *App, boxID string) services.Box {
t.Helper()
var box services.Box
for i := 0; i < 50; i++ {
next, err := app.uploadService.GetBox(boxID)
if err != nil {
t.Fatalf("GetBox returned error: %v", err)
}
box = next
processing := false
for _, file := range box.Files {
if file.Processing {
processing = true
break
}
}
if !processing {
return box
}
time.Sleep(10 * time.Millisecond)
}
t.Fatalf("box %s was still processing: %+v", boxID, box.Files)
return box
}
func copyDir(src, dst string) error {
return filepath.WalkDir(src, func(path string, d os.DirEntry, err error) error {
if err != nil {

View File

@@ -22,6 +22,14 @@ func newCleanupJob(cfg config.Config, logger *slog.Logger, uploadService *servic
if cleaned > 0 {
logger.Info("cleanup job complete", "source", "housekeeping", "severity", "user_activity", "code", 2202, "cleaned", cleaned)
}
cleanedUploads, err := uploadService.CleanupExpiredResumableSessions(time.Now().UTC())
if err != nil {
logger.Warn("resumable upload cleanup failed", "source", "housekeeping", "severity", "warn", "code", 4204, "error", err.Error())
return
}
if cleanedUploads > 0 {
logger.Info("resumable uploads cleaned", "source", "housekeeping", "severity", "user_activity", "code", 2204, "cleaned", cleanedUploads)
}
if banService != nil {
cleanedEvents, err := banService.CleanupAbuseEvents(time.Now().UTC())
if err != nil {
@@ -37,7 +45,12 @@ func newCleanupJob(cfg config.Config, logger *slog.Logger, uploadService *servic
}
func RunCleanupNow(uploadService *services.UploadService, logger *slog.Logger) (int, error) {
return cleanupUnavailableBoxes(uploadService, logger)
cleaned, err := cleanupUnavailableBoxes(uploadService, logger)
if err != nil {
return cleaned, err
}
cleanedUploads, err := uploadService.CleanupExpiredResumableSessions(time.Now().UTC())
return cleaned + cleanedUploads, err
}
func cleanupUnavailableBoxes(uploadService *services.UploadService, logger *slog.Logger) (int, error) {

View File

@@ -0,0 +1,735 @@
package services
import (
"context"
"crypto/sha256"
"crypto/subtle"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"sort"
"strings"
"time"
"go.etcd.io/bbolt"
)
var resumableUploadsBucket = []byte("resumable_uploads")
const (
ResumableStatusUploading = "uploading"
ResumableStatusProcessing = "processing"
ResumableStatusCompleted = "completed"
ResumableStatusCancelled = "cancelled"
)
type ResumableFileInput struct {
Name string `json:"name"`
Size int64 `json:"size"`
ContentType string `json:"contentType"`
Fingerprint string `json:"fingerprint,omitempty"`
}
type ResumableSession struct {
ID string `json:"id"`
Options UploadOptions `json:"options"`
Files []ResumableFile `json:"files"`
ChunkSize int64 `json:"chunkSize"`
Status string `json:"status"`
BoxID string `json:"boxId,omitempty"`
ResumeTokenHash string `json:"resumeTokenHash,omitempty"`
ResumeToken string `json:"-"`
ChunkRoot string `json:"chunkRoot,omitempty"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
ExpiresAt time.Time `json:"expiresAt"`
}
type ResumableFile struct {
ID string `json:"id"`
Name string `json:"name"`
Size int64 `json:"size"`
ContentType string `json:"contentType"`
Fingerprint string `json:"fingerprint,omitempty"`
ChunkCount int `json:"chunkCount"`
UploadedChunks []int `json:"uploadedChunks"`
}
func (s *UploadService) ensureResumableBucket() error {
return s.db.Update(func(tx *bbolt.Tx) error {
_, err := tx.CreateBucketIfNotExists(resumableUploadsBucket)
return err
})
}
func (s *UploadService) CreateResumableSession(files []ResumableFileInput, opts UploadOptions, chunkSize int64, retention time.Duration, chunkRoot string) (ResumableSession, error) {
if len(files) == 0 {
return ResumableSession{}, fmt.Errorf("no files were uploaded")
}
if chunkSize <= 0 {
return ResumableSession{}, fmt.Errorf("chunk size must be positive")
}
if retention <= 0 {
return ResumableSession{}, fmt.Errorf("retention must be positive")
}
if strings.TrimSpace(opts.Password) != "" {
opts.PasswordSalt, opts.PasswordHash = hashPassword(opts.Password)
opts.Password = ""
}
sessionFiles, err := s.resumableFilesFromInput(files, opts, chunkSize, nil)
if err != nil {
return ResumableSession{}, err
}
now := time.Now().UTC()
resumeToken := randomID(32)
sessionID := randomID(12)
session := ResumableSession{
ID: sessionID,
Options: opts,
Files: sessionFiles,
ChunkSize: chunkSize,
Status: ResumableStatusUploading,
ResumeTokenHash: resumableTokenHash(sessionID, resumeToken),
ResumeToken: resumeToken,
ChunkRoot: strings.TrimSpace(chunkRoot),
CreatedAt: now,
UpdatedAt: now,
ExpiresAt: now.Add(retention),
}
if err := s.saveResumableSession(session); err != nil {
return ResumableSession{}, err
}
return session, nil
}
func (s *UploadService) VerifyResumableToken(session ResumableSession, token string) bool {
if session.ResumeTokenHash == "" || strings.TrimSpace(token) == "" {
return false
}
hash := resumableTokenHash(session.ID, token)
return subtle.ConstantTimeCompare([]byte(hash), []byte(session.ResumeTokenHash)) == 1
}
func (s *UploadService) AddResumableFiles(sessionID string, files []ResumableFileInput) (ResumableSession, error) {
if len(files) == 0 {
return s.GetResumableSession(sessionID)
}
session, err := s.GetResumableSession(sessionID)
if err != nil {
return ResumableSession{}, err
}
if err := resumableSessionWritable(session); err != nil {
return ResumableSession{}, err
}
existing := make(map[string]bool)
for _, file := range session.Files {
existing[resumableFileKey(file.Name, file.Size, file.Fingerprint)] = true
}
newFiles, err := s.resumableFilesFromInput(files, session.Options, session.ChunkSize, existing)
if err != nil {
return ResumableSession{}, err
}
if len(newFiles) == 0 {
return session, nil
}
session.Files = append(session.Files, newFiles...)
session.UpdatedAt = time.Now().UTC()
if err := s.saveResumableSession(session); err != nil {
return ResumableSession{}, err
}
return session, nil
}
func (s *UploadService) GetResumableSession(id string) (ResumableSession, error) {
var session ResumableSession
err := s.db.View(func(tx *bbolt.Tx) error {
bucket := tx.Bucket(resumableUploadsBucket)
if bucket == nil {
return os.ErrNotExist
}
data := bucket.Get([]byte(id))
if data == nil {
return os.ErrNotExist
}
return json.Unmarshal(data, &session)
})
if err != nil {
return ResumableSession{}, err
}
return session, nil
}
func (s *UploadService) PutResumableChunk(ctx context.Context, sessionID, fileID string, index int, body io.Reader) (ResumableSession, error) {
session, err := s.GetResumableSession(sessionID)
if err != nil {
return ResumableSession{}, err
}
if err := resumableSessionWritable(session); err != nil {
return ResumableSession{}, err
}
fileIndex := -1
for i, file := range session.Files {
if file.ID == fileID {
fileIndex = i
break
}
}
if fileIndex < 0 {
return ResumableSession{}, os.ErrNotExist
}
file := session.Files[fileIndex]
if index < 0 || index >= file.ChunkCount {
return ResumableSession{}, fmt.Errorf("chunk index is invalid")
}
expectedSize := expectedChunkSize(file.Size, session.ChunkSize, index)
chunkDir := s.resumableFileDirFor(session, file.ID)
if err := os.MkdirAll(chunkDir, 0o755); err != nil {
return ResumableSession{}, err
}
chunkPath := s.resumableChunkPathFor(session, file.ID, index)
tempPath := chunkPath + ".tmp"
target, err := os.OpenFile(tempPath, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
if err != nil {
return ResumableSession{}, err
}
written, copyErr := io.Copy(target, io.LimitReader(body, expectedSize+1))
closeErr := target.Close()
if copyErr != nil {
_ = os.Remove(tempPath)
return ResumableSession{}, copyErr
}
if closeErr != nil {
_ = os.Remove(tempPath)
return ResumableSession{}, closeErr
}
if written != expectedSize {
_ = os.Remove(tempPath)
return ResumableSession{}, fmt.Errorf("chunk size mismatch")
}
if err := os.Rename(tempPath, chunkPath); err != nil {
_ = os.Remove(tempPath)
return ResumableSession{}, err
}
session.Files[fileIndex].UploadedChunks = addChunkIndex(session.Files[fileIndex].UploadedChunks, index)
session.UpdatedAt = time.Now().UTC()
if err := s.saveResumableSession(session); err != nil {
return ResumableSession{}, err
}
return session, nil
}
func (s *UploadService) CompleteResumableSession(ctx context.Context, sessionID string) (UploadResult, ResumableSession, error) {
session, err := s.GetResumableSession(sessionID)
if err != nil {
return UploadResult{}, ResumableSession{}, err
}
if (session.Status == ResumableStatusCompleted || session.Status == ResumableStatusProcessing) && session.BoxID != "" {
box, err := s.GetBox(session.BoxID)
if err != nil {
return UploadResult{}, ResumableSession{}, err
}
return s.resultForBox(box, ""), session, nil
}
if err := resumableSessionWritable(session); err != nil {
return UploadResult{}, ResumableSession{}, err
}
staged, err := s.resumableIncomingFiles(session)
if err != nil {
return UploadResult{}, ResumableSession{}, err
}
result, err := s.CreateBoxFromIncomingContext(ctx, staged, session.Options)
if err != nil {
return UploadResult{}, ResumableSession{}, err
}
if err := os.RemoveAll(s.resumableSessionDirFor(session)); err != nil {
return UploadResult{}, ResumableSession{}, err
}
session.Status = ResumableStatusCompleted
session.BoxID = result.BoxID
session.UpdatedAt = time.Now().UTC()
if err := s.saveResumableSession(session); err != nil {
return UploadResult{}, ResumableSession{}, err
}
return result, session, nil
}
func (s *UploadService) CreateProcessingBoxFromResumable(sessionID string) (UploadResult, ResumableSession, error) {
session, err := s.GetResumableSession(sessionID)
if err != nil {
return UploadResult{}, ResumableSession{}, err
}
if (session.Status == ResumableStatusCompleted || session.Status == ResumableStatusProcessing) && session.BoxID != "" {
box, err := s.GetBox(session.BoxID)
if err != nil {
return UploadResult{}, ResumableSession{}, err
}
return s.resultForBox(box, ""), session, nil
}
if err := resumableSessionWritable(session); err != nil {
return UploadResult{}, ResumableSession{}, err
}
if _, err := s.resumableIncomingFiles(session); err != nil {
return UploadResult{}, ResumableSession{}, err
}
now := time.Now().UTC()
expiresAt := now.AddDate(0, 0, 7)
if session.Options.ExpiresInMinutes < 0 || session.Options.MaxDays < 0 {
expiresAt = now.AddDate(100, 0, 0)
} else if session.Options.ExpiresInMinutes > 0 {
expiresAt = now.Add(time.Duration(session.Options.ExpiresInMinutes) * time.Minute)
} else if session.Options.MaxDays > 0 {
expiresAt = now.Add(time.Duration(session.Options.MaxDays) * 24 * time.Hour)
}
box := Box{
ID: randomID(10),
OwnerID: strings.TrimSpace(session.Options.OwnerID),
CollectionID: strings.TrimSpace(session.Options.CollectionID),
CreatorIP: strings.TrimSpace(session.Options.CreatorIP),
StorageBackendID: normalizeBackendID(session.Options.StorageBackendID),
CreatedAt: now,
ExpiresAt: expiresAt,
MaxDownloads: session.Options.MaxDownloads,
Obfuscate: session.Options.ObfuscateMetadata && (strings.TrimSpace(session.Options.Password) != "" || strings.TrimSpace(session.Options.PasswordHash) != ""),
Files: make([]File, 0, len(session.Files)),
}
deleteToken := randomID(32)
box.DeleteTokenHash = deleteTokenHash(box.ID, deleteToken)
if strings.TrimSpace(session.Options.PasswordHash) != "" {
box.PasswordSalt = session.Options.PasswordSalt
box.PasswordHash = session.Options.PasswordHash
} else if strings.TrimSpace(session.Options.Password) != "" {
salt, hash := hashPassword(session.Options.Password)
box.PasswordSalt = salt
box.PasswordHash = hash
}
for _, incoming := range session.Files {
fileID := randomID(8)
storedName := "@each@" + fileID + strings.ToLower(filepath.Ext(incoming.Name))
objectKey := boxObjectKey(box.ID, storedName)
contentType := incoming.ContentType
if contentType == "" {
contentType = "application/octet-stream"
}
box.Files = append(box.Files, File{
ID: fileID,
Name: filepath.Base(incoming.Name),
StoredName: storedName,
Size: incoming.Size,
ContentType: contentType,
PreviewKind: previewKind(contentType),
ObjectKey: objectKey,
Processing: true,
UploadedAt: now,
})
}
if err := s.saveBoxRecord(box); err != nil {
return UploadResult{}, ResumableSession{}, err
}
session.Status = ResumableStatusProcessing
session.BoxID = box.ID
session.UpdatedAt = time.Now().UTC()
if err := s.saveResumableSession(session); err != nil {
return UploadResult{}, ResumableSession{}, err
}
return s.resultForBox(box, deleteToken), session, nil
}
func (s *UploadService) FinalizeProcessingResumableSession(ctx context.Context, sessionID string) (UploadResult, error) {
session, err := s.GetResumableSession(sessionID)
if err != nil {
return UploadResult{}, err
}
if session.Status == ResumableStatusCompleted && session.BoxID != "" {
box, err := s.GetBox(session.BoxID)
if err != nil {
return UploadResult{}, err
}
return s.resultForBox(box, ""), nil
}
if session.Status != ResumableStatusProcessing || session.BoxID == "" {
return UploadResult{}, fmt.Errorf("upload session is not processing")
}
box, err := s.GetBox(session.BoxID)
if err != nil {
return UploadResult{}, err
}
staged, err := s.resumableIncomingFiles(session)
if err != nil {
return UploadResult{}, err
}
if len(staged) != len(box.Files) {
return UploadResult{}, fmt.Errorf("processing file count mismatch")
}
backend, err := s.storage.Backend(box.StorageBackendID)
if err != nil {
return UploadResult{}, err
}
for i, incoming := range staged {
source, err := incoming.Open()
if err != nil {
return UploadResult{}, err
}
file := box.Files[i]
if err := s.writeUploadedObject(ctx, backend, file.ObjectKey, source, incoming.Size(), 0, incoming.ContentType()); err != nil {
source.Close()
_ = backend.Delete(context.Background(), file.ObjectKey)
box.Files[i].ProcessingError = err.Error()
_ = s.saveBoxRecord(box)
return UploadResult{}, err
}
source.Close()
box.Files[i].Processing = false
box.Files[i].ProcessingError = ""
box.Files[i].UploadedAt = time.Now().UTC()
if err := s.saveBoxRecord(box); err != nil {
return UploadResult{}, err
}
}
if err := s.writeBoxMetadata(box); err != nil {
s.logger.Warn("box metadata write failed after resumable processing", "source", "storage", "severity", "warn", "code", 4020, "box_id", box.ID, "error", err.Error())
}
if err := os.RemoveAll(s.resumableSessionDirFor(session)); err != nil {
return UploadResult{}, err
}
session.Status = ResumableStatusCompleted
session.UpdatedAt = time.Now().UTC()
if err := s.saveResumableSession(session); err != nil {
return UploadResult{}, err
}
return s.resultForBox(box, ""), nil
}
func (s *UploadService) CompleteUploadedResumableSession(ctx context.Context, sessionID string) (UploadResult, ResumableSession, error) {
session, err := s.GetResumableSession(sessionID)
if err != nil {
return UploadResult{}, ResumableSession{}, err
}
if err := resumableSessionWritable(session); err != nil {
return UploadResult{}, ResumableSession{}, err
}
completeFiles := make([]ResumableFile, 0, len(session.Files))
for _, file := range session.Files {
if resumableFileComplete(file) {
completeFiles = append(completeFiles, file)
}
}
if len(completeFiles) == 0 {
return UploadResult{}, ResumableSession{}, fmt.Errorf("no fully uploaded files to finish")
}
partial := session
partial.Files = completeFiles
staged, err := s.resumableIncomingFiles(partial)
if err != nil {
return UploadResult{}, ResumableSession{}, err
}
result, err := s.CreateBoxFromIncomingContext(ctx, staged, session.Options)
if err != nil {
return UploadResult{}, ResumableSession{}, err
}
if err := os.RemoveAll(s.resumableSessionDirFor(session)); err != nil {
return UploadResult{}, ResumableSession{}, err
}
session.Status = ResumableStatusCompleted
session.BoxID = result.BoxID
session.Files = completeFiles
session.UpdatedAt = time.Now().UTC()
if err := s.deleteResumableSession(session.ID); err != nil {
return UploadResult{}, ResumableSession{}, err
}
return result, session, nil
}
func (s *UploadService) CancelResumableSession(sessionID string) error {
session, err := s.GetResumableSession(sessionID)
if err != nil {
return err
}
if err := os.RemoveAll(s.resumableSessionDirFor(session)); err != nil {
return err
}
return s.deleteResumableSession(session.ID)
}
func (s *UploadService) CleanupExpiredResumableSessions(now time.Time) (int, error) {
candidates := make([]ResumableSession, 0)
err := s.db.View(func(tx *bbolt.Tx) error {
bucket := tx.Bucket(resumableUploadsBucket)
if bucket == nil {
return nil
}
return bucket.ForEach(func(_, value []byte) error {
var session ResumableSession
if err := json.Unmarshal(value, &session); err != nil {
return err
}
if session.Status == ResumableStatusCompleted ||
session.Status == ResumableStatusCancelled ||
(session.Status == ResumableStatusUploading && !session.ExpiresAt.After(now)) {
candidates = append(candidates, session)
}
return nil
})
})
if err != nil {
return 0, err
}
for _, session := range candidates {
if err := os.RemoveAll(s.resumableSessionDirFor(session)); err != nil {
return 0, err
}
}
err = s.db.Update(func(tx *bbolt.Tx) error {
bucket := tx.Bucket(resumableUploadsBucket)
if bucket == nil {
return nil
}
for _, session := range candidates {
if err := bucket.Delete([]byte(session.ID)); err != nil {
return err
}
}
return nil
})
return len(candidates), err
}
func (s *UploadService) deleteResumableSession(sessionID string) error {
return s.db.Update(func(tx *bbolt.Tx) error {
bucket := tx.Bucket(resumableUploadsBucket)
if bucket == nil {
return nil
}
return bucket.Delete([]byte(sessionID))
})
}
func (s *UploadService) saveResumableSession(session ResumableSession) error {
if err := s.ensureResumableBucket(); err != nil {
return err
}
return s.db.Update(func(tx *bbolt.Tx) error {
data, err := json.Marshal(session)
if err != nil {
return err
}
return tx.Bucket(resumableUploadsBucket).Put([]byte(session.ID), data)
})
}
func (s *UploadService) resumableFilesFromInput(files []ResumableFileInput, opts UploadOptions, chunkSize int64, existing map[string]bool) ([]ResumableFile, error) {
sessionFiles := make([]ResumableFile, 0, len(files))
for _, file := range files {
file.Name = filepath.Base(strings.TrimSpace(file.Name))
if file.Name == "." || file.Name == "" {
return nil, fmt.Errorf("file name is required")
}
if file.Size < 0 {
return nil, fmt.Errorf("file size is invalid")
}
fingerprint := strings.TrimSpace(file.Fingerprint)
key := resumableFileKey(file.Name, file.Size, fingerprint)
if existing != nil && existing[key] {
continue
}
if !opts.SkipSizeLimit {
if err := s.ValidateSize(file.Size); err != nil {
return nil, err
}
}
chunks := int((file.Size + chunkSize - 1) / chunkSize)
if chunks == 0 {
chunks = 1
}
sessionFiles = append(sessionFiles, ResumableFile{
ID: randomID(8),
Name: file.Name,
Size: file.Size,
ContentType: strings.TrimSpace(file.ContentType),
Fingerprint: fingerprint,
ChunkCount: chunks,
})
if existing != nil {
existing[key] = true
}
}
return sessionFiles, nil
}
func resumableFileKey(name string, size int64, fingerprint string) string {
return strings.TrimSpace(fingerprint) + "|" + filepath.Base(strings.TrimSpace(name)) + "|" + fmt.Sprintf("%d", size)
}
type resumableIncomingFile struct {
service *UploadService
session ResumableSession
file ResumableFile
}
func (f resumableIncomingFile) Name() string {
return f.file.Name
}
func (f resumableIncomingFile) Size() int64 {
return f.file.Size
}
func (f resumableIncomingFile) ContentType() string {
return f.file.ContentType
}
func (f resumableIncomingFile) Open() (io.ReadCloser, error) {
return &resumableChunkReader{
service: f.service,
session: f.session,
file: f.file,
}, nil
}
type resumableChunkReader struct {
service *UploadService
session ResumableSession
file ResumableFile
index int
current *os.File
}
func (r *resumableChunkReader) Read(p []byte) (int, error) {
for {
if r.current == nil {
if r.index >= r.file.ChunkCount {
return 0, io.EOF
}
chunk, err := os.Open(r.service.resumableChunkPathFor(r.session, r.file.ID, r.index))
if err != nil {
return 0, err
}
r.current = chunk
}
n, err := r.current.Read(p)
if err == io.EOF {
if closeErr := r.current.Close(); closeErr != nil {
r.current = nil
return n, closeErr
}
r.current = nil
r.index++
if n > 0 {
return n, nil
}
continue
}
return n, err
}
}
func (r *resumableChunkReader) Close() error {
if r.current == nil {
return nil
}
err := r.current.Close()
r.current = nil
return err
}
func (s *UploadService) resumableIncomingFiles(session ResumableSession) ([]IncomingFile, error) {
staged := make([]IncomingFile, 0, len(session.Files))
for _, file := range session.Files {
if len(file.UploadedChunks) != file.ChunkCount {
return nil, fmt.Errorf("file %s is missing chunks", file.Name)
}
var written int64
for i := 0; i < file.ChunkCount; i++ {
info, err := os.Stat(s.resumableChunkPathFor(session, file.ID, i))
if err != nil {
return nil, fmt.Errorf("file %s is missing chunks", file.Name)
}
written += info.Size()
}
if written != file.Size {
return nil, fmt.Errorf("chunk size total mismatch")
}
staged = append(staged, resumableIncomingFile{
service: s,
session: session,
file: file,
})
}
return staged, nil
}
func resumableSessionWritable(session ResumableSession) error {
if session.Status != ResumableStatusUploading {
return fmt.Errorf("upload session is not active")
}
if !session.ExpiresAt.After(time.Now().UTC()) {
return fmt.Errorf("upload session expired")
}
return nil
}
func resumableFileComplete(file ResumableFile) bool {
return file.ChunkCount > 0 && len(file.UploadedChunks) == file.ChunkCount
}
func expectedChunkSize(fileSize, chunkSize int64, index int) int64 {
offset := int64(index) * chunkSize
remaining := fileSize - offset
if remaining < 0 {
return 0
}
if remaining > chunkSize {
return chunkSize
}
return remaining
}
func addChunkIndex(chunks []int, index int) []int {
for _, chunk := range chunks {
if chunk == index {
return chunks
}
}
chunks = append(chunks, index)
sort.Ints(chunks)
return chunks
}
func resumableTokenHash(sessionID, token string) string {
sum := sha256.Sum256([]byte("warpbox-resumable:" + sessionID + ":" + token))
return hex.EncodeToString(sum[:])
}
func (s *UploadService) resumableSessionDir(sessionID string) string {
return filepath.Join(s.dataDir, "tmp", "uploads", sessionID)
}
func (s *UploadService) resumableSessionDirFor(session ResumableSession) string {
if strings.TrimSpace(session.ChunkRoot) != "" {
return filepath.Join(session.ChunkRoot, session.ID)
}
return s.resumableSessionDir(session.ID)
}
func (s *UploadService) resumableFileDir(sessionID, fileID string) string {
return filepath.Join(s.resumableSessionDir(sessionID), fileID)
}
func (s *UploadService) resumableFileDirFor(session ResumableSession, fileID string) string {
return filepath.Join(s.resumableSessionDirFor(session), fileID)
}
func (s *UploadService) resumableChunkPath(sessionID, fileID string, index int) string {
return filepath.Join(s.resumableFileDir(sessionID, fileID), fmt.Sprintf("%06d.part", index))
}
func (s *UploadService) resumableChunkPathFor(session ResumableSession, fileID string, index int) string {
return filepath.Join(s.resumableFileDirFor(session, fileID), fmt.Sprintf("%06d.part", index))
}

View File

@@ -37,6 +37,11 @@ type UploadPolicySettings struct {
ShortWindowSeconds int `json:"shortWindowSeconds"`
AnonymousStorageBackend string `json:"anonymousStorageBackend"`
UserStorageBackend string `json:"userStorageBackend"`
ResumableUploadsEnabled bool `json:"resumableUploadsEnabled"`
ResumableChunkSizeMB float64 `json:"resumableChunkSizeMb"`
ResumableRetentionHours int `json:"resumableRetentionHours"`
ResumableChunkMode string `json:"resumableChunkMode"`
ResumableChunkPath string `json:"resumableChunkPath"`
}
type UsageRecord struct {
@@ -89,6 +94,11 @@ func NewSettingsService(db *bbolt.DB, defaults config.SettingsDefaults) (*Settin
ShortWindowSeconds: defaults.ShortWindowSeconds,
AnonymousStorageBackend: defaults.AnonymousStorageBackend,
UserStorageBackend: defaults.UserStorageBackend,
ResumableUploadsEnabled: defaults.ResumableUploadsEnabled,
ResumableChunkSizeMB: defaults.ResumableChunkSizeMB,
ResumableRetentionHours: defaults.ResumableRetentionHours,
ResumableChunkMode: defaults.ResumableChunkMode,
ResumableChunkPath: defaults.ResumableChunkPath,
},
}
service.defaults = service.withBuiltinDefaultGaps(service.defaults)
@@ -143,6 +153,15 @@ func (s *SettingsService) withBuiltinDefaultGaps(settings UploadPolicySettings)
if strings.TrimSpace(settings.UserStorageBackend) == "" {
settings.UserStorageBackend = StorageBackendLocal
}
if settings.ResumableChunkSizeMB <= 0 {
settings.ResumableChunkSizeMB = 8
}
if settings.ResumableRetentionHours <= 0 {
settings.ResumableRetentionHours = 24
}
if strings.TrimSpace(settings.ResumableChunkMode) == "" {
settings.ResumableChunkMode = "same"
}
return settings
}
@@ -156,6 +175,13 @@ func (s *SettingsService) UploadPolicy() (UploadPolicySettings, error) {
if err := json.Unmarshal(data, &settings); err != nil {
return err
}
var raw map[string]json.RawMessage
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
if _, ok := raw["resumableUploadsEnabled"]; !ok {
settings.ResumableUploadsEnabled = s.defaults.ResumableUploadsEnabled
}
settings = s.withDefaultGaps(settings)
return nil
})
@@ -217,6 +243,15 @@ func (s *SettingsService) withDefaultGaps(settings UploadPolicySettings) UploadP
if strings.TrimSpace(settings.UserStorageBackend) == "" {
settings.UserStorageBackend = s.defaults.UserStorageBackend
}
if settings.ResumableChunkSizeMB <= 0 {
settings.ResumableChunkSizeMB = s.defaults.ResumableChunkSizeMB
}
if settings.ResumableRetentionHours <= 0 {
settings.ResumableRetentionHours = s.defaults.ResumableRetentionHours
}
if strings.TrimSpace(settings.ResumableChunkMode) == "" {
settings.ResumableChunkMode = s.defaults.ResumableChunkMode
}
return settings
}
@@ -422,6 +457,18 @@ func (s *SettingsService) validate(settings UploadPolicySettings) error {
if settings.ShortWindowRequests <= 0 || settings.ShortWindowSeconds <= 0 {
return fmt.Errorf("short-window rate limits must be positive")
}
if settings.ResumableChunkSizeMB <= 0 {
return fmt.Errorf("resumable chunk size must be positive")
}
if settings.ResumableRetentionHours <= 0 {
return fmt.Errorf("resumable retention must be positive")
}
if settings.ResumableChunkMode != "same" && settings.ResumableChunkMode != "custom" {
return fmt.Errorf("resumable chunk storage mode is invalid")
}
if settings.ResumableChunkMode == "custom" && strings.TrimSpace(settings.ResumableChunkPath) == "" {
return fmt.Errorf("custom resumable chunk path is required")
}
return nil
}

View File

@@ -42,6 +42,8 @@ type UploadOptions struct {
ExpiresInMinutes int
MaxDownloads int
Password string
PasswordSalt string
PasswordHash string
ObfuscateMetadata bool
OwnerID string
CollectionID string
@@ -50,6 +52,56 @@ type UploadOptions struct {
StorageBackendID string
}
type IncomingFile interface {
Name() string
Size() int64
ContentType() string
Open() (io.ReadCloser, error)
}
type multipartIncomingFile struct {
header *multipart.FileHeader
}
func (f multipartIncomingFile) Name() string {
return f.header.Filename
}
func (f multipartIncomingFile) Size() int64 {
return f.header.Size
}
func (f multipartIncomingFile) ContentType() string {
return f.header.Header.Get("Content-Type")
}
func (f multipartIncomingFile) Open() (io.ReadCloser, error) {
return f.header.Open()
}
type StagedUploadFile struct {
Filename string
FileSize int64
MIMEType string
Path string
}
func (f StagedUploadFile) Name() string {
return f.Filename
}
func (f StagedUploadFile) Size() int64 {
return f.FileSize
}
func (f StagedUploadFile) ContentType() string {
return f.MIMEType
}
func (f StagedUploadFile) Open() (io.ReadCloser, error) {
return os.Open(f.Path)
}
type Box struct {
ID string `json:"id"`
OwnerID string `json:"ownerId,omitempty"`
@@ -78,6 +130,8 @@ type File struct {
Thumbnail string `json:"thumbnail,omitempty"`
ObjectKey string `json:"objectKey,omitempty"`
ThumbnailObjectKey string `json:"thumbnailObjectKey,omitempty"`
Processing bool `json:"processing,omitempty"`
ProcessingError string `json:"processingError,omitempty"`
UploadedAt time.Time `json:"uploadedAt"`
}
@@ -98,6 +152,7 @@ type ResultFile struct {
Size string `json:"size"`
URL string `json:"url"`
ThumbnailURL string `json:"thumbnailUrl"`
Processing bool `json:"processing,omitempty"`
}
type AdminStats struct {
@@ -198,6 +253,14 @@ func (s *UploadService) ValidateSize(size int64) error {
}
func (s *UploadService) CreateBox(files []*multipart.FileHeader, opts UploadOptions) (UploadResult, error) {
return s.CreateBoxFromIncoming(multipartIncomingFiles(files), opts)
}
func (s *UploadService) CreateBoxFromIncoming(files []IncomingFile, opts UploadOptions) (UploadResult, error) {
return s.CreateBoxFromIncomingContext(context.Background(), files, opts)
}
func (s *UploadService) CreateBoxFromIncomingContext(ctx context.Context, files []IncomingFile, opts UploadOptions) (UploadResult, error) {
if len(files) == 0 {
return UploadResult{}, fmt.Errorf("no files were uploaded")
}
@@ -232,13 +295,16 @@ func (s *UploadService) CreateBox(files []*multipart.FileHeader, opts UploadOpti
}
deleteToken := randomID(32)
box.DeleteTokenHash = deleteTokenHash(box.ID, deleteToken)
if strings.TrimSpace(opts.Password) != "" {
if strings.TrimSpace(opts.PasswordHash) != "" {
box.PasswordSalt = opts.PasswordSalt
box.PasswordHash = opts.PasswordHash
} else if strings.TrimSpace(opts.Password) != "" {
salt, hash := hashPassword(opts.Password)
box.PasswordSalt = salt
box.PasswordHash = hash
}
if err := s.writeFilesToBox(&box, files, opts); err != nil {
if err := s.writeIncomingFilesToBox(ctx, &box, files, opts); err != nil {
return UploadResult{}, err
}
@@ -261,6 +327,10 @@ func (s *UploadService) CreateBox(files []*multipart.FileHeader, opts UploadOpti
// selection into a single box). The box keeps its original expiry, password and
// other settings; only the new files are written.
func (s *UploadService) AppendFiles(boxID string, files []*multipart.FileHeader, opts UploadOptions) (UploadResult, error) {
return s.AppendIncomingFiles(boxID, multipartIncomingFiles(files), opts)
}
func (s *UploadService) AppendIncomingFiles(boxID string, files []IncomingFile, opts UploadOptions) (UploadResult, error) {
if len(files) == 0 {
return UploadResult{}, fmt.Errorf("no files were uploaded")
}
@@ -268,7 +338,7 @@ func (s *UploadService) AppendFiles(boxID string, files []*multipart.FileHeader,
if err != nil {
return UploadResult{}, err
}
if err := s.writeFilesToBox(&box, files, opts); err != nil {
if err := s.writeIncomingFilesToBox(context.Background(), &box, files, opts); err != nil {
return UploadResult{}, err
}
if err := s.SaveBox(box); err != nil {
@@ -289,14 +359,26 @@ func (s *UploadService) AppendFiles(boxID string, files []*multipart.FileHeader,
// appends the file metadata to box.Files. The box's StorageBackendID determines
// where files land, so it works for both new and existing boxes.
func (s *UploadService) writeFilesToBox(box *Box, files []*multipart.FileHeader, opts UploadOptions) error {
return s.writeIncomingFilesToBox(context.Background(), box, multipartIncomingFiles(files), opts)
}
func multipartIncomingFiles(files []*multipart.FileHeader) []IncomingFile {
incoming := make([]IncomingFile, 0, len(files))
for _, file := range files {
incoming = append(incoming, multipartIncomingFile{header: file})
}
return incoming
}
func (s *UploadService) writeIncomingFilesToBox(ctx context.Context, box *Box, files []IncomingFile, opts UploadOptions) error {
backend, err := s.storage.Backend(box.StorageBackendID)
if err != nil {
return err
}
for _, header := range files {
for _, incoming := range files {
if !opts.SkipSizeLimit {
if err := s.ValidateSize(header.Size); err != nil {
if err := s.ValidateSize(incoming.Size()); err != nil {
return err
}
}
@@ -306,15 +388,15 @@ func (s *UploadService) writeFilesToBox(box *Box, files []*multipart.FileHeader,
maxSize = 0
}
file, err := header.Open()
file, err := incoming.Open()
if err != nil {
return err
}
fileID := randomID(8)
storedName := "@each@" + fileID + strings.ToLower(filepath.Ext(header.Filename))
storedName := "@each@" + fileID + strings.ToLower(filepath.Ext(incoming.Name()))
objectKey := boxObjectKey(box.ID, storedName)
contentType := header.Header.Get("Content-Type")
contentType := incoming.ContentType()
if contentType == "" {
buffer := make([]byte, 512)
n, _ := file.Read(buffer)
@@ -324,17 +406,18 @@ func (s *UploadService) writeFilesToBox(box *Box, files []*multipart.FileHeader,
}
}
if err := s.writeUploadedObject(context.Background(), backend, objectKey, file, header.Size, maxSize, contentType); err != nil {
if err := s.writeUploadedObject(ctx, backend, objectKey, file, incoming.Size(), maxSize, contentType); err != nil {
file.Close()
_ = backend.Delete(context.Background(), objectKey)
return err
}
file.Close()
box.Files = append(box.Files, File{
ID: fileID,
Name: filepath.Base(header.Filename),
Name: filepath.Base(incoming.Name()),
StoredName: storedName,
Size: header.Size,
Size: incoming.Size(),
ContentType: contentType,
PreviewKind: previewKind(contentType),
ObjectKey: objectKey,
@@ -736,6 +819,9 @@ func (s *UploadService) ThumbnailObjectKey(box Box, file File) string {
}
func (s *UploadService) OpenFileObject(ctx context.Context, box Box, file File) (StorageObject, error) {
if file.Processing {
return StorageObject{}, fmt.Errorf("file is still processing")
}
backend, err := s.storage.Backend(s.BoxStorageBackendID(box))
if err != nil {
return StorageObject{}, err
@@ -857,6 +943,13 @@ func (s *UploadService) WriteZip(w io.Writer, box Box) error {
}
func (s *UploadService) SaveBox(box Box) error {
if err := s.saveBoxRecord(box); err != nil {
return err
}
return s.writeBoxMetadata(box)
}
func (s *UploadService) saveBoxRecord(box Box) error {
if box.StorageBackendID == "" {
box.StorageBackendID = StorageBackendLocal
}
@@ -866,10 +959,7 @@ func (s *UploadService) SaveBox(box Box) error {
}
return s.db.Update(func(tx *bbolt.Tx) error {
if err := tx.Bucket(boxesBucket).Put([]byte(box.ID), data); err != nil {
return err
}
return s.writeBoxMetadata(box)
return tx.Bucket(boxesBucket).Put([]byte(box.ID), data)
})
}
@@ -882,6 +972,7 @@ func (s *UploadService) resultForBox(box Box, deleteToken string) UploadResult {
Size: helpers.FormatBytes(file.Size),
URL: fmt.Sprintf("%s/d/%s/f/%s", s.baseURL, box.ID, file.ID),
ThumbnailURL: fmt.Sprintf("%s/d/%s/thumb/%s", s.baseURL, box.ID, file.ID),
Processing: file.Processing,
})
}
@@ -931,21 +1022,34 @@ func writeUploadedFile(path string, source multipart.File, maxSize int64) error
return nil
}
func (s *UploadService) writeUploadedObject(ctx context.Context, backend StorageBackend, key string, source multipart.File, size, maxSize int64, contentType string) error {
func (s *UploadService) writeUploadedObject(ctx context.Context, backend StorageBackend, key string, source io.Reader, size, maxSize int64, contentType string) error {
var reader io.Reader = source
putSize := size
if maxSize > 0 {
reader = io.LimitReader(source, maxSize+1)
var buffer bytes.Buffer
written, err := io.Copy(&buffer, reader)
if err != nil {
return err
}
if written > maxSize {
if size > maxSize {
return fmt.Errorf("file exceeds max upload size")
}
return backend.Put(ctx, key, bytes.NewReader(buffer.Bytes()), written, contentType)
reader = io.LimitReader(source, maxSize)
putSize = size
}
if ctx != nil {
reader = contextReader{ctx: ctx, reader: reader}
}
return backend.Put(ctx, key, reader, putSize, contentType)
}
type contextReader struct {
ctx context.Context
reader io.Reader
}
func (r contextReader) Read(p []byte) (int, error) {
select {
case <-r.ctx.Done():
return 0, r.ctx.Err()
default:
return r.reader.Read(p)
}
return backend.Put(ctx, key, reader, size, contentType)
}
func boxObjectKey(boxID, name string) string {

View File

@@ -126,6 +126,262 @@ func TestLocalStorageBackendAndLegacyFallback(t *testing.T) {
object.Body.Close()
}
func TestResumableSessionUploadOutOfOrderAndComplete(t *testing.T) {
service := newTestUploadService(t)
session, err := service.CreateResumableSession([]ResumableFileInput{{
Name: "note.txt",
Size: 11,
ContentType: "text/plain",
Fingerprint: "sha256:first-chunk",
}}, UploadOptions{MaxDays: 1, Password: "secret"}, 4, time.Hour, "")
if err != nil {
t.Fatalf("CreateResumableSession returned error: %v", err)
}
if session.ResumeToken == "" || session.ResumeTokenHash == "" {
t.Fatalf("resumable session did not create resume token: %+v", session)
}
if !service.VerifyResumableToken(session, session.ResumeToken) {
t.Fatalf("VerifyResumableToken rejected correct token")
}
if service.VerifyResumableToken(session, "wrong-token") {
t.Fatalf("VerifyResumableToken accepted wrong token")
}
stored, err := service.GetResumableSession(session.ID)
if err != nil {
t.Fatalf("GetResumableSession returned error: %v", err)
}
if stored.ResumeToken != "" {
t.Fatalf("stored session leaked raw resume token")
}
if strings.Contains(stored.ResumeTokenHash, session.ResumeToken) {
t.Fatalf("stored token hash contains raw token")
}
if !service.VerifyResumableToken(stored, session.ResumeToken) {
t.Fatalf("stored session rejected correct token")
}
if session.Options.Password != "" || session.Options.PasswordHash == "" || session.Options.PasswordSalt == "" {
t.Fatalf("resumable session did not hash password before storage: %+v", session.Options)
}
if session.Files[0].ChunkCount != 3 {
t.Fatalf("ChunkCount = %d, want 3", session.Files[0].ChunkCount)
}
if session.Files[0].Fingerprint != "sha256:first-chunk" {
t.Fatalf("Fingerprint = %q", session.Files[0].Fingerprint)
}
for index, body := range map[int]string{2: "rld", 0: "hell", 1: "o wo"} {
updated, err := service.PutResumableChunk(testContext(), session.ID, session.Files[0].ID, index, strings.NewReader(body))
if err != nil {
t.Fatalf("PutResumableChunk(%d) returned error: %v", index, err)
}
if len(updated.Files[0].UploadedChunks) == 0 {
t.Fatalf("UploadedChunks was not updated")
}
}
result, completed, err := service.CompleteResumableSession(testContext(), session.ID)
if err != nil {
t.Fatalf("CompleteResumableSession returned error: %v", err)
}
if completed.Status != ResumableStatusCompleted || completed.BoxID != result.BoxID {
t.Fatalf("completed session = %+v, result = %+v", completed, result)
}
box := getTestBox(t, service, result.BoxID)
if box.PasswordHash == "" || box.PasswordSalt == "" || box.PasswordHash != session.Options.PasswordHash {
t.Fatalf("completed box did not preserve hashed password")
}
object, err := service.OpenFileObject(testContext(), box, box.Files[0])
if err != nil {
t.Fatalf("OpenFileObject returned error: %v", err)
}
data, err := io.ReadAll(object.Body)
object.Body.Close()
if err != nil {
t.Fatalf("ReadAll returned error: %v", err)
}
if string(data) != "hello world" {
t.Fatalf("object body = %q", string(data))
}
if _, err := os.Stat(service.resumableSessionDir(session.ID)); !os.IsNotExist(err) {
t.Fatalf("resumable temp dir after complete error = %v, want os.ErrNotExist", err)
}
replayed, replayedSession, err := service.CompleteResumableSession(testContext(), session.ID)
if err != nil {
t.Fatalf("CompleteResumableSession replay returned error: %v", err)
}
if replayed.BoxID != result.BoxID || replayedSession.Status != ResumableStatusCompleted {
t.Fatalf("replayed result = %+v, session = %+v, want box %s completed", replayed, replayedSession, result.BoxID)
}
}
func TestResumableCompleteRejectsMissingChunks(t *testing.T) {
service := newTestUploadService(t)
session, err := service.CreateResumableSession([]ResumableFileInput{{
Name: "note.txt",
Size: 8,
ContentType: "text/plain",
}}, UploadOptions{MaxDays: 1}, 4, time.Hour, "")
if err != nil {
t.Fatalf("CreateResumableSession returned error: %v", err)
}
if _, err := service.PutResumableChunk(testContext(), session.ID, session.Files[0].ID, 0, strings.NewReader("hell")); err != nil {
t.Fatalf("PutResumableChunk returned error: %v", err)
}
if _, _, err := service.CompleteResumableSession(testContext(), session.ID); err == nil {
t.Fatalf("CompleteResumableSession accepted missing chunks")
}
}
func TestResumablePartialCompleteKeepsOnlyFinishedFiles(t *testing.T) {
service := newTestUploadService(t)
session, err := service.CreateResumableSession([]ResumableFileInput{
{Name: "done.txt", Size: 4, ContentType: "text/plain", Fingerprint: "done"},
{Name: "partial.txt", Size: 8, ContentType: "text/plain", Fingerprint: "partial"},
}, UploadOptions{MaxDays: 1}, 4, time.Hour, "")
if err != nil {
t.Fatalf("CreateResumableSession returned error: %v", err)
}
if _, err := service.PutResumableChunk(testContext(), session.ID, session.Files[0].ID, 0, strings.NewReader("done")); err != nil {
t.Fatalf("PutResumableChunk done returned error: %v", err)
}
if _, err := service.PutResumableChunk(testContext(), session.ID, session.Files[1].ID, 0, strings.NewReader("part")); err != nil {
t.Fatalf("PutResumableChunk partial returned error: %v", err)
}
result, completed, err := service.CompleteUploadedResumableSession(testContext(), session.ID)
if err != nil {
t.Fatalf("CompleteUploadedResumableSession returned error: %v", err)
}
if completed.Status != ResumableStatusCompleted || completed.BoxID != result.BoxID || len(completed.Files) != 1 {
t.Fatalf("completed session = %+v, result = %+v", completed, result)
}
box := getTestBox(t, service, result.BoxID)
if len(box.Files) != 1 || box.Files[0].Name != "done.txt" {
t.Fatalf("partial completion box files = %+v", box.Files)
}
object, err := service.OpenFileObject(testContext(), box, box.Files[0])
if err != nil {
t.Fatalf("OpenFileObject returned error: %v", err)
}
data, err := io.ReadAll(object.Body)
object.Body.Close()
if err != nil {
t.Fatalf("ReadAll returned error: %v", err)
}
if string(data) != "done" {
t.Fatalf("partial completion object = %q", string(data))
}
if _, err := service.GetResumableSession(session.ID); !os.IsNotExist(err) {
t.Fatalf("GetResumableSession after partial complete error = %v, want os.ErrNotExist", err)
}
if _, err := os.Stat(service.resumableSessionDir(session.ID)); !os.IsNotExist(err) {
t.Fatalf("resumable temp dir after partial complete error = %v, want os.ErrNotExist", err)
}
}
func TestResumablePartialCompleteRejectsNoFinishedFiles(t *testing.T) {
service := newTestUploadService(t)
session, err := service.CreateResumableSession([]ResumableFileInput{{
Name: "partial.txt",
Size: 8,
ContentType: "text/plain",
}}, UploadOptions{MaxDays: 1}, 4, time.Hour, "")
if err != nil {
t.Fatalf("CreateResumableSession returned error: %v", err)
}
if _, err := service.PutResumableChunk(testContext(), session.ID, session.Files[0].ID, 0, strings.NewReader("part")); err != nil {
t.Fatalf("PutResumableChunk returned error: %v", err)
}
if _, _, err := service.CompleteUploadedResumableSession(testContext(), session.ID); err == nil {
t.Fatalf("CompleteUploadedResumableSession accepted no completed files")
}
if _, err := service.GetResumableSession(session.ID); err != nil {
t.Fatalf("GetResumableSession after failed partial complete returned error: %v", err)
}
}
func TestResumableSessionCanAddFilesBeforeComplete(t *testing.T) {
service := newTestUploadService(t)
session, err := service.CreateResumableSession([]ResumableFileInput{{
Name: "one.txt",
Size: 4,
ContentType: "text/plain",
Fingerprint: "one",
}}, UploadOptions{MaxDays: 1}, 4, time.Hour, "")
if err != nil {
t.Fatalf("CreateResumableSession returned error: %v", err)
}
if _, err := service.PutResumableChunk(testContext(), session.ID, session.Files[0].ID, 0, strings.NewReader("one!")); err != nil {
t.Fatalf("PutResumableChunk one returned error: %v", err)
}
updated, err := service.AddResumableFiles(session.ID, []ResumableFileInput{{
Name: "two.txt",
Size: 4,
ContentType: "text/plain",
Fingerprint: "two",
}})
if err != nil {
t.Fatalf("AddResumableFiles returned error: %v", err)
}
if len(updated.Files) != 2 {
t.Fatalf("files after add = %d, want 2", len(updated.Files))
}
if updated.Files[0].UploadedChunks[0] != 0 {
t.Fatalf("existing uploaded chunk was not preserved: %+v", updated.Files[0])
}
if _, err := service.AddResumableFiles(session.ID, []ResumableFileInput{{
Name: "two.txt",
Size: 4,
ContentType: "text/plain",
Fingerprint: "two",
}}); err != nil {
t.Fatalf("duplicate AddResumableFiles returned error: %v", err)
}
updated, err = service.GetResumableSession(session.ID)
if err != nil {
t.Fatalf("GetResumableSession returned error: %v", err)
}
if len(updated.Files) != 2 {
t.Fatalf("duplicate add changed file count to %d", len(updated.Files))
}
if _, err := service.PutResumableChunk(testContext(), session.ID, updated.Files[1].ID, 0, strings.NewReader("two!")); err != nil {
t.Fatalf("PutResumableChunk two returned error: %v", err)
}
result, _, err := service.CompleteResumableSession(testContext(), session.ID)
if err != nil {
t.Fatalf("CompleteResumableSession returned error: %v", err)
}
box := getTestBox(t, service, result.BoxID)
if len(box.Files) != 2 {
t.Fatalf("completed box file count = %d, want 2", len(box.Files))
}
}
func TestResumableCleanupRemovesExpiredSessionsAndChunks(t *testing.T) {
service := newTestUploadService(t)
session, err := service.CreateResumableSession([]ResumableFileInput{{
Name: "note.txt",
Size: 4,
ContentType: "text/plain",
}}, UploadOptions{MaxDays: 1}, 4, time.Millisecond, "")
if err != nil {
t.Fatalf("CreateResumableSession returned error: %v", err)
}
if _, err := service.PutResumableChunk(testContext(), session.ID, session.Files[0].ID, 0, strings.NewReader("hell")); err != nil {
t.Fatalf("PutResumableChunk returned error: %v", err)
}
cleaned, err := service.CleanupExpiredResumableSessions(time.Now().UTC().Add(time.Hour))
if err != nil {
t.Fatalf("CleanupExpiredResumableSessions returned error: %v", err)
}
if cleaned != 1 {
t.Fatalf("cleaned = %d, want 1", cleaned)
}
if _, err := service.GetResumableSession(session.ID); !os.IsNotExist(err) {
t.Fatalf("GetResumableSession after cleanup error = %v, want os.ErrNotExist", err)
}
if _, err := os.Stat(service.resumableSessionDir(session.ID)); !os.IsNotExist(err) {
t.Fatalf("resumable temp dir after cleanup error = %v, want os.ErrNotExist", err)
}
}
func TestContaboStorageConfigAllowsDisplayNamesWithSpaces(t *testing.T) {
service := newTestUploadService(t)
cfg, err := service.Storage().CreateS3Backend(StorageBackendConfig{

View File

@@ -48,6 +48,10 @@
width: 100%;
}
.upload-options .form-footer .upload-new-button {
margin-top: -0.25rem;
}
.hero-copy {
text-align: center;
}
@@ -335,10 +339,13 @@ button {
.file-progress-side {
width: min(10rem, 32vw);
display: grid;
grid-template-columns: minmax(0, 1fr) auto;
gap: 0.35rem;
align-items: center;
}
.file-progress-percent {
grid-column: 1 / -1;
color: var(--muted-foreground);
font-size: 0.75rem;
text-align: right;
@@ -349,6 +356,81 @@ button {
margin-top: 0;
}
.upload-file-remove {
width: 1.65rem;
height: 1.65rem;
min-height: 1.65rem;
padding: 0;
border-color: var(--border);
border-radius: 999px;
color: var(--muted-foreground);
background: var(--surface-1);
font-size: 1rem;
line-height: 1;
}
.upload-file-remove:hover {
color: var(--foreground);
border-color: var(--primary);
background: var(--surface-1-hover);
}
.upload-file-waiting {
border-color: color-mix(in srgb, var(--primary) 42%, var(--border));
background: color-mix(in srgb, var(--primary) 8%, var(--background));
}
.upload-file-complete .file-progress span {
background: var(--success);
}
.upload-file-state {
grid-column: 1 / -1;
color: var(--muted-foreground);
font-size: 0.72rem;
text-align: right;
}
.upload-recovery-overlay {
position: fixed;
inset: 0;
z-index: 80;
display: grid;
place-items: center;
padding: 1rem;
background: color-mix(in srgb, var(--background) 72%, transparent);
backdrop-filter: blur(10px);
}
.upload-recovery-modal {
width: min(34rem, 100%);
box-shadow: var(--shadow-lg);
}
.upload-recovery-modal h2 {
margin: 0 0 0.65rem;
font-size: 1.35rem;
}
.upload-recovery-modal p {
margin: 0;
color: var(--muted-foreground);
line-height: 1.55;
}
.upload-recovery-actions {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 0.7rem;
margin-top: 1.25rem;
}
@media (max-width: 640px) {
.upload-recovery-actions {
grid-template-columns: 1fr;
}
}
.result-item small,
.download-item small,
.result-item code,

View File

@@ -46,6 +46,15 @@
text-decoration: none;
}
.upload-processing-alert {
margin: 1rem 0;
padding: .85rem 1rem;
border: 1px solid color-mix(in srgb, var(--primary) 45%, transparent);
border-radius: var(--radius);
background: color-mix(in srgb, var(--primary) 12%, transparent);
color: var(--foreground);
}
.thumb-link {
flex: 0 0 4.75rem;
width: 4.75rem;
@@ -200,6 +209,15 @@
padding: 0;
}
.file-card.is-processing {
opacity: .62;
filter: grayscale(.25);
}
.file-card.is-processing .file-open {
cursor: wait;
}
.file-reaction-dock {
position: static;
z-index: 2;

View File

@@ -5,6 +5,17 @@
window.open(url, "_blank", "noopener,noreferrer");
};
window.Warpbox.absoluteURL = function absoluteURL(url) {
if (!url) {
return "";
}
try {
return new URL(url, window.location.origin).href;
} catch (_) {
return url;
}
};
window.Warpbox.writeClipboard = async function writeClipboard(text) {
if (navigator.clipboard && window.isSecureContext) {
await navigator.clipboard.writeText(text);
@@ -26,6 +37,9 @@
if (!text || !button) {
return;
}
if (typeof text === "string" && (text.startsWith("/") || /^https?:\/\//i.test(text))) {
text = window.Warpbox.absoluteURL(text);
}
await window.Warpbox.writeClipboard(text);
const previous = button.textContent;
button.textContent = copiedLabel;

View File

@@ -124,7 +124,7 @@
}
async function copyPreviewLink(button) {
await window.Warpbox.writeClipboard(button.href);
await window.Warpbox.writeClipboard(window.Warpbox.absoluteURL(button.href));
const label = button.querySelector("[data-preview-label]");
if (!label) {
return;
@@ -164,11 +164,11 @@
return true;
}
if (action === "copy-preview") {
await window.Warpbox.writeClipboard(file.previewURL);
await window.Warpbox.writeClipboard(window.Warpbox.absoluteURL(file.previewURL));
return true;
}
if (action === "copy-download") {
await window.Warpbox.writeClipboard(file.downloadURL);
await window.Warpbox.writeClipboard(window.Warpbox.absoluteURL(file.downloadURL));
return true;
}
if (action === "download") {

View File

@@ -13,6 +13,8 @@
const copyURL = document.querySelector("#copy-url");
const openBox = document.querySelector("#open-box");
const manageLink = document.querySelector("#manage-link");
const newUpload = document.querySelector("#new-upload");
const RESUMABLE_SESSIONS_KEY = "warpbox-resumable-sessions";
if (!form || !dropZone || !fileInput) {
return;
@@ -42,6 +44,9 @@
let latestBoxURL = "";
let selectedFiles = [];
let uploadLocked = false;
let recoveredDraft = null;
let resumeMode = false;
["dragenter", "dragover"].forEach((eventName) => {
dropZone.addEventListener(eventName, (event) => {
@@ -57,33 +62,67 @@
});
});
dropZone.addEventListener("drop", (event) => {
if (event.dataTransfer && event.dataTransfer.files.length > 0) {
fileInput.files = event.dataTransfer.files;
updateSelectedState(event.dataTransfer.files);
document.addEventListener("dragover", (event) => {
if (event.dataTransfer && Array.from(event.dataTransfer.types || []).includes("Files")) {
event.preventDefault();
}
});
fileInput.addEventListener("change", () => updateSelectedState(fileInput.files));
document.addEventListener("drop", (event) => {
if (!event.dataTransfer || !event.dataTransfer.files.length) {
return;
}
event.preventDefault();
if (!dropZone.contains(event.target)) {
addSelectedFiles(event.dataTransfer.files);
}
});
dropZone.addEventListener("drop", (event) => {
if (event.dataTransfer && event.dataTransfer.files.length > 0) {
addSelectedFiles(event.dataTransfer.files);
}
});
fileInput.addEventListener("change", () => {
addSelectedFiles(fileInput.files);
fileInput.value = "";
});
form.addEventListener("submit", async (event) => {
event.preventDefault();
if (!fileInput.files || fileInput.files.length === 0) {
if (selectedFiles.length === 0) {
updateStatus("Choose at least one file first.");
return;
}
const submit = form.querySelector("button[type='submit']");
const formData = new FormData(form);
selectedFiles = Array.from(fileInput.files);
renderQueue(selectedFiles, "queued");
const formData = uploadFormData();
if (resumeMode && recoveredDraft) {
renderResumeQueue(recoveredDraft.session, selectedFiles);
} else {
renderQueue(selectedFiles, "queued");
}
setLoading(true, submit);
try {
const payload = await uploadWithProgress(form.action, formData, selectedFiles);
const payload = await uploadResumable(form.action, formData, selectedFiles);
renderResult(payload);
form.reset();
updateSelectedState([]);
selectedFiles = [];
resumeMode = false;
recoveredDraft = null;
fileInput.value = "";
if (uploadQueue) {
uploadQueue.hidden = true;
uploadQueue.replaceChildren();
}
if (newUpload) {
newUpload.hidden = true;
}
if (fileSummary) {
fileSummary.textContent = "Upload complete.";
}
} catch (error) {
updateStatus(error.message || "Upload failed");
} finally {
@@ -97,25 +136,66 @@
});
}
function updateSelectedState(files) {
selectedFiles = Array.from(files || []);
if (newUpload) {
newUpload.addEventListener("click", () => {
cancelRecoveredDraft().catch((error) => {
updateStatus(error.message || "Upload draft could not be deleted");
});
});
}
recoverResumableSessions();
function addSelectedFiles(files) {
if (uploadLocked) {
return;
}
Array.from(files || []).forEach((file) => {
if (!selectedFiles.some((existing) => fileIdentity(existing) === fileIdentity(file))) {
selectedFiles.push(file);
}
});
updateSelectedState();
}
function removeSelectedFile(index) {
if (uploadLocked) {
return;
}
selectedFiles.splice(index, 1);
updateSelectedState();
}
function updateSelectedState() {
const count = selectedFiles.length || 0;
const title = dropZone.querySelector(".drop-title");
if (title) {
title.textContent = count === 0 ? "Drop files to upload" : count === 1 ? "1 file selected" : `${count} files selected`;
}
if (fileSummary) {
fileSummary.textContent = count === 0 ? "Choose one or more files to begin." : `${count} file${count === 1 ? "" : "s"} ready.`;
if (resumeMode && recoveredDraft) {
fileSummary.textContent = count === 0
? "Reselect missing files to resume, or add extra files to this upload."
: `${count} local file${count === 1 ? "" : "s"} ready for the recovered upload.`;
} else {
fileSummary.textContent = count === 0 ? "Choose one or more files to begin." : `${count} file${count === 1 ? "" : "s"} ready.`;
}
}
if (count > 0) {
if (resumeMode && recoveredDraft) {
renderResumeQueue(recoveredDraft.session, selectedFiles);
} else if (count > 0) {
renderQueue(selectedFiles, "queued");
} else if (uploadQueue) {
uploadQueue.hidden = true;
uploadQueue.replaceChildren();
}
if (newUpload) {
newUpload.hidden = !(resumeMode && recoveredDraft);
}
}
function setLoading(isLoading, submit) {
uploadLocked = isLoading;
if (progress) {
progress.hidden = !isLoading;
}
@@ -123,6 +203,9 @@
submit.disabled = isLoading;
submit.textContent = isLoading ? "Uploading..." : "Upload files";
}
if (newUpload) {
newUpload.disabled = isLoading;
}
updateStatus(isLoading ? "Transferring files..." : "");
setTotalProgress(isLoading ? 0 : 100);
}
@@ -138,15 +221,15 @@
return;
}
latestBoxURL = payload.boxUrl;
latestBoxURL = window.Warpbox.absoluteURL(payload.boxUrl);
result.hidden = false;
openBox.href = payload.boxUrl;
openBox.href = latestBoxURL;
resultMeta.textContent = `${payload.files.length} file${payload.files.length === 1 ? "" : "s"} · expires ${window.Warpbox.formatDate(payload.expiresAt)}`;
if (manageLink) {
const anchor = manageLink.querySelector("a");
manageLink.hidden = !payload.manageUrl;
if (anchor && payload.manageUrl) {
anchor.href = payload.manageUrl;
anchor.href = window.Warpbox.absoluteURL(payload.manageUrl);
}
}
@@ -154,11 +237,12 @@
payload.files.forEach((file) => {
resultList.append(createFileRow({
name: file.name,
meta: `${file.size} · ${file.url}`,
meta: `${file.size} · ${window.Warpbox.absoluteURL(file.url)}`,
progress: 100,
status: "complete",
}));
});
result.scrollIntoView({ behavior: "smooth", block: "start" });
}
function uploadWithProgress(url, formData, files) {
@@ -201,26 +285,612 @@
});
}
async function uploadResumable(fallbackUrl, formData, files) {
if (!window.fetch || typeof Blob === "undefined") {
return uploadWithProgress(fallbackUrl, formData, files);
}
updateStatus("Fingerprinting files...");
const fingerprints = await Promise.all(files.map((file) => fileFingerprint(file)));
const createPayload = {
files: files.map((file, index) => ({
name: file.name,
size: file.size,
contentType: file.type || "application/octet-stream",
fingerprint: fingerprints[index],
})),
expiresMinutes: parseInt(formData.get("expires_minutes") || "0", 10) || 0,
maxDownloads: parseInt(formData.get("max_downloads") || "0", 10) || 0,
password: formData.get("password") || "",
obfuscateMetadata: formData.get("obfuscate_metadata") === "on",
collectionId: formData.get("collection_id") || "",
};
const persistable = !createPayload.password;
let session = null;
if (persistable && resumeMode && recoveredDraft) {
session = await fetchResumableStatus(recoveredDraft.session.sessionId, recoveredDraft.session.resumeToken);
session.resumeToken = recoveredDraft.session.resumeToken;
} else if (persistable) {
session = await findResumableSession(createPayload);
}
if (session) {
validateResumeSelection(session, createPayload);
session = await addMissingResumableFiles(session, createPayload);
if (resumeMode && recoveredDraft && recoveredDraft.session.sessionId === session.sessionId) {
recoveredDraft.session = session;
}
if (persistable) {
saveResumableSession(session, createPayload);
}
}
if (!session || session.status !== "uploading") {
try {
session = await createResumableSession(createPayload);
} catch (error) {
if ((error.message || "").toLowerCase().includes("resumable uploads are disabled")) {
return uploadWithProgress(fallbackUrl, formData, files);
}
throw error;
}
if (persistable) {
saveResumableSession(session, createPayload);
}
}
const sessionFiles = files.map((file, index) => matchSessionFile(session, createPayload.files[index]));
if (sessionFiles.some((file) => !file)) {
throw new Error("Upload session could not match the selected files");
}
updateStatus("Uploading...");
const totalBytes = files.reduce((sum, file) => sum + file.size, 0);
const completedByFile = new Array(files.length).fill(0);
sessionFiles.forEach((sessionFile, index) => {
completedByFile[index] = uploadedBytesForSessionFile(sessionFile, session.chunkSize);
setSingleFileProgress(index, files[index], percentForBytes(completedByFile[index], files[index].size));
});
setTotalProgress(percentForBytes(completedByFile.reduce((sum, bytes) => sum + bytes, 0), totalBytes));
for (let fileIndex = 0; fileIndex < files.length; fileIndex++) {
const file = files[fileIndex];
const sessionFile = sessionFiles[fileIndex];
const uploaded = new Set(sessionFile.uploadedChunks || []);
for (let chunkIndex = 0; chunkIndex < sessionFile.chunkCount; chunkIndex++) {
if (uploaded.has(chunkIndex)) {
continue;
}
const start = chunkIndex * session.chunkSize;
const end = Math.min(file.size, start + session.chunkSize);
await uploadChunkWithRetry(session, sessionFile, chunkIndex, file.slice(start, end), (loaded) => {
const currentTotal = completedByFile.reduce((sum, bytes) => sum + bytes, 0) + loaded;
setTotalProgress(percentForBytes(currentTotal, totalBytes));
setSingleFileProgress(fileIndex, file, percentForBytes(completedByFile[fileIndex] + loaded, file.size));
updateStatus(`${percentForBytes(currentTotal, totalBytes)}%`);
});
completedByFile[fileIndex] += end - start;
uploaded.add(chunkIndex);
sessionFile.uploadedChunks = Array.from(uploaded).sort((a, b) => a - b);
if (persistable) {
saveResumableSession(session, createPayload);
}
}
setSingleFileProgress(fileIndex, file, 100);
}
updateStatus("Finalizing upload...");
const resultPayload = await completeResumableSession(session.sessionId, session.resumeToken);
const wasResumeMode = resumeMode;
if (persistable) {
removeResumableSession(session.sessionId);
}
if (resumeMode && recoveredDraft && recoveredDraft.session.sessionId === session.sessionId) {
resumeMode = false;
recoveredDraft = null;
}
setTotalProgress(100);
if (!wasResumeMode) {
setFileProgress(files, 100);
}
return resultPayload;
}
async function createResumableSession(payload) {
const response = await fetch("/api/v1/uploads/resumable", {
method: "POST",
headers: {
"Accept": "application/json",
"Content-Type": "application/json",
},
body: JSON.stringify(payload),
});
return readUploadJSON(response, "Upload session could not be created");
}
async function fetchResumableStatus(sessionID, resumeToken) {
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}`, {
headers: resumableHeaders(resumeToken),
});
return readUploadJSON(response, "Upload session could not be resumed");
}
async function addResumableFiles(sessionID, resumeToken, files) {
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}/files`, {
method: "POST",
headers: {
...resumableHeaders(resumeToken),
"Accept": "application/json",
"Content-Type": "application/json",
},
body: JSON.stringify({ files }),
});
return readUploadJSON(response, "Upload session files could not be added");
}
function uploadChunk(sessionID, resumeToken, fileID, chunkIndex, chunk, onProgress) {
return new Promise((resolve, reject) => {
const request = new XMLHttpRequest();
request.open("PUT", `/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}/files/${encodeURIComponent(fileID)}/chunks/${chunkIndex}`);
request.setRequestHeader("Accept", "application/json");
request.setRequestHeader("X-Warpbox-Resume-Token", resumeToken || "");
request.upload.addEventListener("progress", (event) => {
if (event.lengthComputable && onProgress) {
onProgress(event.loaded);
}
});
request.addEventListener("load", () => {
if (request.status < 200 || request.status >= 300) {
let payload = {};
try {
payload = JSON.parse(request.responseText || "{}");
} catch (error) {
payload = {};
}
reject(new Error(payload.error || "Chunk upload failed"));
return;
}
resolve();
});
request.addEventListener("error", () => reject(new Error("Network error during chunk upload")));
request.addEventListener("abort", () => reject(new Error("Chunk upload aborted")));
request.send(chunk);
});
}
async function uploadChunkWithRetry(session, sessionFile, chunkIndex, chunk, onProgress) {
const delays = [1000, 2000, 5000, 10000, 20000];
let lastError = null;
for (let attempt = 0; attempt <= delays.length; attempt++) {
try {
return await uploadChunk(session.sessionId, session.resumeToken, sessionFile.id, chunkIndex, chunk, onProgress);
} catch (error) {
lastError = error;
if (attempt >= delays.length) {
break;
}
const seconds = Math.round(delays[attempt] / 1000);
updateStatus(`Connection interrupted, retrying chunk ${chunkIndex + 1} in ${seconds}s`);
await wait(delays[attempt]);
}
}
throw lastError || new Error("Chunk upload failed");
}
async function completeResumableSession(sessionID, resumeToken) {
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}/complete`, {
method: "POST",
headers: resumableHeaders(resumeToken),
});
return readUploadJSON(response, "Upload could not be completed");
}
async function cancelResumableSession(sessionID, resumeToken) {
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}`, {
method: "DELETE",
headers: resumableHeaders(resumeToken),
});
if (!response.ok && response.status !== 404) {
await readUploadJSON(response, "Upload draft could not be deleted");
}
}
function resumableHeaders(resumeToken) {
return {
"Accept": "application/json",
"X-Warpbox-Resume-Token": resumeToken || "",
};
}
function wait(ms) {
return new Promise((resolve) => window.setTimeout(resolve, ms));
}
async function readUploadJSON(response, fallback) {
let payload = {};
try {
payload = await response.json();
} catch (error) {
payload = {};
}
if (!response.ok) {
throw new Error(payload.error || fallback);
}
return payload;
}
async function findResumableSession(payload) {
const records = loadResumableSessions();
const optionKey = resumableOptionKey(payload);
const selectedKeys = new Set(payload.files.map((file) => resumableFileKey(file)));
for (const record of records) {
if (record.optionKey !== optionKey) {
continue;
}
if (!record.files || !record.files.some((file) => selectedKeys.has(resumableFileKey(file)))) {
continue;
}
const session = await fetchResumableStatus(record.sessionId, record.resumeToken).catch(() => null);
if (!session || session.status !== "uploading") {
removeResumableSession(record.sessionId);
continue;
}
session.resumeToken = record.resumeToken;
const sessionKeys = new Set(session.files.map((file) => resumableFileKey(file)));
const selectedContainsSessionFile = Array.from(sessionKeys).some((key) => selectedKeys.has(key));
if (selectedContainsSessionFile) {
return session;
}
}
return null;
}
async function addMissingResumableFiles(session, payload) {
const existing = new Set(session.files.map((file) => resumableFileKey(file)));
const missing = payload.files.filter((file) => !existing.has(resumableFileKey(file)));
if (missing.length === 0) {
return session;
}
const updated = await addResumableFiles(session.sessionId, session.resumeToken, missing);
updated.resumeToken = session.resumeToken;
return updated;
}
function validateResumeSelection(session, payload) {
if (!resumeMode || !recoveredDraft || session.sessionId !== recoveredDraft.session.sessionId) {
return;
}
const existingByNameSize = new Map();
(session.files || []).forEach((file) => {
existingByNameSize.set(`${file.name}:${file.size}`, resumableFileKey(file));
});
for (const file of payload.files || []) {
const expectedKey = existingByNameSize.get(`${file.name}:${file.size}`);
if (expectedKey && expectedKey !== resumableFileKey(file)) {
throw new Error(`"${file.name}" does not match the pending upload. Select the exact original file.`);
}
}
}
function matchSessionFile(session, file) {
const key = resumableFileKey(file);
return session.files.find((sessionFile) => resumableFileKey(sessionFile) === key) || null;
}
function resumableOptionKey(payload) {
return [
payload.expiresMinutes,
payload.maxDownloads,
payload.obfuscateMetadata ? "1" : "0",
payload.collectionId || "",
].join(":");
}
function resumableFileKey(file) {
return [file.name, file.size, file.fingerprint || ""].join(":");
}
function loadResumableSessions() {
try {
const value = localStorage.getItem(RESUMABLE_SESSIONS_KEY);
const records = value ? JSON.parse(value) : [];
return Array.isArray(records) ? records : [];
} catch (error) {
return [];
}
}
function saveResumableSession(session, payload) {
try {
const records = loadResumableSessions().filter((record) => record.sessionId !== session.sessionId);
records.push({
sessionId: session.sessionId,
resumeToken: session.resumeToken || "",
optionKey: resumableOptionKey(payload),
options: {
expiresMinutes: payload.expiresMinutes,
maxDownloads: payload.maxDownloads,
obfuscateMetadata: !!payload.obfuscateMetadata,
collectionId: payload.collectionId || "",
},
files: session.files.map((file) => ({
name: file.name,
size: file.size,
contentType: file.contentType || "application/octet-stream",
fingerprint: file.fingerprint || "",
uploadedChunks: file.uploadedChunks || [],
chunkCount: file.chunkCount || 0,
})),
updatedAt: new Date().toISOString(),
});
localStorage.setItem(RESUMABLE_SESSIONS_KEY, JSON.stringify(records.slice(-25)));
} catch (error) {
/* ignore persistence failures */
}
}
async function recoverResumableSessions() {
const records = loadResumableSessions()
.filter((record) => record.sessionId && record.resumeToken)
.sort((a, b) => new Date(b.updatedAt || 0).getTime() - new Date(a.updatedAt || 0).getTime());
if (records.length === 0) {
return;
}
for (const record of records) {
const session = await fetchResumableStatus(record.sessionId, record.resumeToken).catch(() => null);
if (!session || session.status !== "uploading") {
removeResumableSession(record.sessionId);
continue;
}
session.resumeToken = record.resumeToken;
recoveredDraft = { session, record };
selectedFiles = [];
renderRecoveredQueue([{ session, record }]);
updateRecoveredSummary(session);
showRecoveryModal(recoveredDraft);
return;
}
}
function updateRecoveredSummary(session) {
updateStatus("Unfinished upload found. Choose how to continue.");
if (fileSummary) {
const totalFiles = (session.files || []).length;
const completedFiles = completedSessionFiles(session).length;
fileSummary.textContent = `Recovered ${totalFiles} pending file${totalFiles === 1 ? "" : "s"}; ${completedFiles} fully uploaded.`;
}
}
function removeResumableSession(sessionID) {
try {
const records = loadResumableSessions().filter((record) => record.sessionId !== sessionID);
localStorage.setItem(RESUMABLE_SESSIONS_KEY, JSON.stringify(records));
} catch (error) {
/* ignore persistence failures */
}
}
function completedSessionFiles(session) {
return (session.files || []).filter((file) => (file.uploadedChunks || []).length >= file.chunkCount);
}
function showRecoveryModal(draft) {
const old = document.querySelector(".upload-recovery-overlay");
if (old) {
old.remove();
}
const completeCount = completedSessionFiles(draft.session).length;
const totalCount = (draft.session.files || []).length;
const overlay = document.createElement("div");
overlay.className = "upload-recovery-overlay";
overlay.setAttribute("role", "dialog");
overlay.setAttribute("aria-modal", "true");
overlay.setAttribute("aria-labelledby", "upload-recovery-title");
const modal = document.createElement("div");
modal.className = "upload-recovery-modal card";
const content = document.createElement("div");
content.className = "card-content";
const title = document.createElement("h2");
title.id = "upload-recovery-title";
title.textContent = "Unfinished upload found";
const copy = document.createElement("p");
copy.textContent = `Warpbox found a private draft with ${totalCount} file${totalCount === 1 ? "" : "s"}. ${completeCount} file${completeCount === 1 ? " is" : "s are"} already fully uploaded.`;
const actions = document.createElement("div");
actions.className = "upload-recovery-actions";
const startOver = document.createElement("button");
startOver.type = "button";
startOver.className = "button button-danger";
startOver.textContent = "New Upload";
startOver.addEventListener("click", async () => {
startOver.disabled = true;
try {
await cancelRecoveredDraft();
overlay.remove();
} catch (error) {
startOver.disabled = false;
updateStatus(error.message || "Upload draft could not be deleted");
}
});
const resume = document.createElement("button");
resume.type = "button";
resume.className = "button button-primary";
resume.textContent = "Resume";
resume.addEventListener("click", () => {
resumeRecoveredDraft();
overlay.remove();
});
actions.append(startOver, resume);
content.append(title, copy, actions);
modal.append(content);
overlay.append(modal);
document.body.append(overlay);
}
async function cancelRecoveredDraft() {
if (!recoveredDraft) {
resetFreshUploadState();
return;
}
const draft = recoveredDraft;
updateStatus("Deleting unfinished upload...");
await cancelResumableSession(draft.session.sessionId, draft.session.resumeToken);
removeResumableSession(draft.session.sessionId);
resetFreshUploadState();
}
function resumeRecoveredDraft() {
if (!recoveredDraft) {
return;
}
resumeMode = true;
selectedFiles = [];
renderResumeQueue(recoveredDraft.session, selectedFiles);
updateSelectedState();
updateStatus("Drop or reselect missing files to continue. Extra files will be added to this upload.");
}
function resetFreshUploadState() {
selectedFiles = [];
resumeMode = false;
recoveredDraft = null;
fileInput.value = "";
result.hidden = true;
if (resultList) {
resultList.replaceChildren();
}
setTotalProgress(0);
updateStatus("");
updateSelectedState();
}
function uploadedBytesForSessionFile(file, chunkSize) {
return (file.uploadedChunks || []).reduce((sum, index) => {
const start = index * chunkSize;
const end = Math.min(file.size, start + chunkSize);
return sum + Math.max(0, end - start);
}, 0);
}
function renderRecoveredQueue(items) {
if (!uploadQueue) {
return;
}
const rows = [];
items.forEach(({ session }) => {
(session.files || []).forEach((file) => {
const uploadedBytes = uploadedBytesForSessionFile(file, session.chunkSize);
const complete = (file.uploadedChunks || []).length >= file.chunkCount;
rows.push({
name: file.name,
size: file.size,
uploadedBytes,
meta: complete
? `${window.Warpbox.formatBytes(file.size)} · uploaded`
: `${window.Warpbox.formatBytes(uploadedBytes)} of ${window.Warpbox.formatBytes(file.size)} · Drop/reselect this file to continue`,
progress: percentForBytes(uploadedBytes, file.size),
status: complete ? "complete" : "waiting",
readonly: true,
});
});
});
uploadQueue.hidden = rows.length === 0;
uploadQueue.replaceChildren();
rows.forEach((row) => uploadQueue.append(createFileRow(row)));
const totalBytes = rows.reduce((sum, row) => sum + (row.size || 0), 0);
if (totalBytes > 0) {
setTotalProgress(percentForBytes(rows.reduce((sum, row) => sum + (row.uploadedBytes || 0), 0), totalBytes));
} else if (rows.length > 0) {
const completed = rows.filter((row) => row.status === "complete").length;
setTotalProgress(percentForBytes(completed, rows.length));
}
}
function renderResumeQueue(session, localFiles) {
if (!uploadQueue) {
return;
}
const rows = [];
const localByNameSize = new Map();
(localFiles || []).forEach((file, index) => {
localByNameSize.set(`${file.name}:${file.size}`, { file, index });
});
const usedLocalIndexes = new Set();
(session.files || []).forEach((file) => {
const uploadedBytes = uploadedBytesForSessionFile(file, session.chunkSize);
const complete = (file.uploadedChunks || []).length >= file.chunkCount;
const localMatch = localByNameSize.get(`${file.name}:${file.size}`) || null;
if (localMatch) {
usedLocalIndexes.add(localMatch.index);
}
rows.push({
name: file.name,
size: file.size,
uploadedBytes,
meta: complete
? `${window.Warpbox.formatBytes(file.size)} · uploaded`
: localMatch
? `${window.Warpbox.formatBytes(uploadedBytes)} of ${window.Warpbox.formatBytes(file.size)} · ready to resume`
: `${window.Warpbox.formatBytes(uploadedBytes)} of ${window.Warpbox.formatBytes(file.size)} · waiting for local file`,
progress: percentForBytes(uploadedBytes, file.size),
status: complete ? "complete" : localMatch ? "queued" : "waiting",
readonly: !localMatch,
index: localMatch ? localMatch.index : undefined,
removable: Boolean(localMatch && !complete),
});
});
(localFiles || []).forEach((file, index) => {
if (usedLocalIndexes.has(index)) {
return;
}
rows.push({
name: file.name,
meta: `${window.Warpbox.formatBytes(file.size)} · new file`,
progress: 0,
status: "queued",
index,
removable: true,
});
});
uploadQueue.hidden = rows.length === 0;
uploadQueue.replaceChildren();
rows.forEach((row) => uploadQueue.append(createFileRow(row)));
}
function percentForBytes(bytes, total) {
if (!total) {
return 100;
}
return Math.max(0, Math.min(100, Math.round((bytes / total) * 100)));
}
function renderQueue(files, status) {
if (!uploadQueue) {
return;
}
uploadQueue.hidden = files.length === 0;
uploadQueue.replaceChildren();
files.forEach((file) => {
files.forEach((file, index) => {
uploadQueue.append(createFileRow({
name: file.name,
meta: window.Warpbox.formatBytes(file.size),
progress: status === "queued" ? 0 : 100,
status,
index,
removable: status === "queued",
}));
});
}
function createFileRow(file) {
const row = document.createElement("div");
row.className = "result-item upload-file-row";
row.className = `result-item upload-file-row upload-file-${file.status || "queued"}`;
row.dataset.fileName = file.name;
if (typeof file.index === "number") {
row.dataset.fileIndex = file.index;
}
const body = document.createElement("span");
const name = document.createElement("strong");
@@ -242,11 +912,53 @@
fill.style.transform = `scaleX(${file.progress / 100})`;
bar.append(fill);
side.append(percent, bar);
if (file.status === "waiting") {
const badge = document.createElement("small");
badge.className = "upload-file-state";
badge.textContent = "Needs local file";
side.append(badge);
}
if (file.removable) {
const remove = document.createElement("button");
remove.className = "upload-file-remove";
remove.type = "button";
remove.setAttribute("aria-label", `Remove ${file.name}`);
remove.textContent = "×";
remove.addEventListener("click", () => removeSelectedFile(file.index || 0));
side.append(remove);
}
row.append(body, side);
return row;
}
function uploadFormData() {
const formData = new FormData(form);
formData.delete("file");
selectedFiles.forEach((file) => {
formData.append("file", file, file.name);
});
return formData;
}
function fileIdentity(file) {
return [file.name, file.size, file.lastModified || 0].join(":");
}
async function fileFingerprint(file) {
if (!window.crypto || !window.crypto.subtle || !file.slice || typeof TextEncoder === "undefined") {
return fileIdentity(file);
}
const sampleSize = Math.min(file.size, 1024 * 1024);
const sample = await file.slice(0, sampleSize).arrayBuffer();
const metadata = new TextEncoder().encode([file.name, file.size, file.lastModified || 0, sampleSize].join(":"));
const combined = new Uint8Array(metadata.byteLength + sample.byteLength);
combined.set(metadata, 0);
combined.set(new Uint8Array(sample), metadata.byteLength);
const digest = await window.crypto.subtle.digest("SHA-256", combined);
return Array.from(new Uint8Array(digest)).map((byte) => byte.toString(16).padStart(2, "0")).join("");
}
function setTotalProgress(percent) {
if (totalProgressBar) {
totalProgressBar.style.transform = `scaleX(${Math.max(0, Math.min(100, percent)) / 100})`;
@@ -271,4 +983,23 @@
}
});
}
function setSingleFileProgress(index, file, progress) {
if (!uploadQueue) {
return;
}
const row = uploadQueue.querySelector(`.upload-file-row[data-file-index="${index}"]`);
if (!row) {
return;
}
const percent = row.querySelector(".file-progress-percent");
const fill = row.querySelector(".file-progress span");
const normalized = Math.max(0, Math.min(100, progress));
if (percent) {
percent.textContent = `${normalized}%`;
}
if (fill) {
fill.style.transform = `scaleX(${normalized / 100})`;
}
}
})();

View File

@@ -126,6 +126,33 @@
</label>
</div>
<div class="settings-section">
<h3 class="settings-section-title">Resumable uploads</h3>
<label class="checkbox-field">
<input type="checkbox" name="resumable_uploads_enabled" {{if .Data.Settings.ResumableUploadsEnabled}}checked{{end}}>
<span>Enable browser/API resumable uploads</span>
</label>
<label>
<span>Chunk size (MB)</span>
<input name="resumable_chunk_size_mb" value="{{.Data.Settings.ResumableChunkSizeMB}}" required>
</label>
<label>
<span>Draft retention (hours)</span>
<input type="number" name="resumable_retention_hours" min="1" value="{{.Data.Settings.ResumableRetentionHours}}" required>
</label>
<label>
<span>Chunk storage</span>
<select name="resumable_chunk_mode" required>
<option value="same" {{if eq .Data.Settings.ResumableChunkMode "same"}}selected{{end}}>Default local data path</option>
<option value="custom" {{if eq .Data.Settings.ResumableChunkMode "custom"}}selected{{end}}>Custom local path, e.g. fast SSD</option>
</select>
</label>
<label>
<span>Custom chunk path</span>
<input name="resumable_chunk_path" value="{{.Data.Settings.ResumableChunkPath}}" placeholder="/mnt/fast-ssd/warpbox-chunks">
</label>
</div>
<button class="button button-primary" type="submit">Save settings</button>
</form>
</div>

View File

@@ -14,6 +14,10 @@
<h2>Endpoints</h2>
<dl class="endpoint-list">
<div><dt>Upload</dt><dd><code>POST /api/v1/upload</code></dd></div>
<div><dt>Resumable create</dt><dd><code>POST /api/v1/uploads/resumable</code></dd></div>
<div><dt>Resumable status</dt><dd><code>GET /api/v1/uploads/resumable/{sessionID}</code></dd></div>
<div><dt>Resumable chunk</dt><dd><code>PUT /api/v1/uploads/resumable/{sessionID}/files/{fileID}/chunks/{index}</code></dd></div>
<div><dt>Resumable complete</dt><dd><code>POST /api/v1/uploads/resumable/{sessionID}/complete</code></dd></div>
<div><dt>Health</dt><dd><code>GET /health</code></dd></div>
<div><dt>Request schema</dt><dd><a href="/api/v1/schemas/upload-request.json"><code>/api/v1/schemas/upload-request.json</code></a></dd></div>
<div><dt>Response schema</dt><dd><a href="/api/v1/schemas/upload-response.json"><code>/api/v1/schemas/upload-response.json</code></a></dd></div>
@@ -21,6 +25,28 @@
</div>
</article>
<article class="card docs-card docs-card-wide">
<div class="card-content">
<h2>Resumable uploads</h2>
<p>Browser uploads use the resumable API by default. Custom clients can use the same flow: create a session with file metadata, upload exact-sized chunks, then complete the session. Chunks are temporary and are cleaned if the session expires.</p>
<pre><code># 1. Create a session.
curl -s {{.Data.BaseURL}}/api/v1/uploads/resumable \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"files":[{"name":"report.pdf","size":1048576,"contentType":"application/pdf"}],"expiresMinutes":1440}'
# 2. Upload each chunk using the returned sessionId, file id, and chunkSize.
dd if=./report.pdf bs=8388608 count=1 skip=0 2>/dev/null | \
curl -X PUT --data-binary @- \
{{.Data.BaseURL}}/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
# 3. Complete after all chunks are present. The response is the normal upload JSON.
curl -X POST -H 'Accept: application/json' \
{{.Data.BaseURL}}/api/v1/uploads/resumable/SESSION_ID/complete</code></pre>
<p class="muted-copy">For authenticated uploads, send the same <code>Authorization: Bearer &lt;token&gt;</code> header on every resumable request. Incomplete chunks are stored under <code>data/tmp/uploads</code> before finalizing into the selected storage backend.</p>
</div>
</article>
<article class="card docs-card">
<div class="card-content">
<h2>Curl upload</h2>

View File

@@ -24,6 +24,12 @@
{{end}}
{{if .Data.Files}}
{{$processing := false}}{{range .Data.Files}}{{if .Processing}}{{$processing = true}}{{end}}{{end}}
{{if $processing}}
<div class="upload-processing-alert" role="status">
Some files are still processing. You can share this link now, but processing files will become available shortly.
</div>
{{end}}
{{$single := eq (len .Data.Files) 1}}
<div class="badge-row">
<span class="badge badge-expiry">Expires {{.Data.ExpiresLabel}}</span>
@@ -74,8 +80,8 @@
</div>
<div class="download-list file-browser is-thumbs" data-file-browser>
{{range .Data.Files}}
<article class="download-item file-card" data-kind="{{.PreviewKind}}" data-file-context data-preview-url="{{.URL}}" data-view-url="{{.DownloadURL}}?inline=1" data-download-url="{{.DownloadURL}}" data-file-name="{{.Name}}" data-reaction-card data-react-url="{{.ReactURL}}" data-reacted="{{if .Reacted}}true{{else}}false{{end}}">
<a class="file-open" href="{{.DownloadURL}}?inline=1"{{if not $single}} target="_blank" rel="noopener noreferrer"{{end}} aria-label="Open {{.Name}}">
<article class="download-item file-card {{if .Processing}}is-processing{{end}}" data-kind="{{.PreviewKind}}" {{if not .Processing}}data-file-context data-preview-url="{{.URL}}" data-view-url="{{.DownloadURL}}?inline=1" data-download-url="{{.DownloadURL}}"{{end}} data-file-name="{{.Name}}" data-reaction-card data-react-url="{{.ReactURL}}" data-reacted="{{if .Reacted}}true{{else}}false{{end}}">
{{if .Processing}}<div class="file-open" aria-label="{{.Name}} is processing">{{else}}<a class="file-open" href="{{.DownloadURL}}?inline=1"{{if not $single}} target="_blank" rel="noopener noreferrer"{{end}} aria-label="Open {{.Name}}">{{end}}
<span class="file-media">
{{if .HasThumbnail}}
<img class="file-thumb" src="{{.ThumbnailURL}}" alt="" loading="lazy">
@@ -86,11 +92,11 @@
</span>
<span class="file-main">
<strong class="file-name" title="{{.Name}}">{{.Name}}</strong>
<small>{{.Size}} · {{.ContentType}}</small>
<small>{{.Size}} · {{if .Processing}}Processing{{else}}{{.ContentType}}{{end}}</small>
</span>
<span class="file-type">{{.ContentType}}</span>
<span class="file-type">{{if .Processing}}Processing{{else}}{{.ContentType}}{{end}}</span>
<span class="file-size">{{.Size}}</span>
</a>
{{if .Processing}}</div>{{else}}</a>{{end}}
{{if not $.Data.Locked}}
<div class="file-reaction-dock" data-reaction-dock>
<div class="file-reactions" data-reaction-list>

View File

@@ -77,6 +77,7 @@
<div class="form-footer">
<p id="file-summary">Choose one or more files to begin.</p>
<button class="button button-primary" type="submit">Upload files</button>
<button class="button button-danger upload-new-button" type="button" id="new-upload" hidden>New upload</button>
</div>
</div>
</div>

View File

@@ -9,6 +9,11 @@ WARPBOX_CLEANUP_ENABLED=true
WARPBOX_CLEANUP_EVERY=1h
WARPBOX_THUMBNAIL_ENABLED=true
WARPBOX_THUMBNAIL_EVERY=1m
WARPBOX_RESUMABLE_UPLOADS_ENABLED=true
WARPBOX_RESUMABLE_CHUNK_MB=16
WARPBOX_RESUMABLE_RETENTION_HOURS=1
WARPBOX_RESUMABLE_CHUNK_MODE=same
WARPBOX_RESUMABLE_CHUNK_PATH=
WARPBOX_MAX_UPLOAD_SIZE_MB=16384
WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true
WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512