feat(upload): add resumable chunk configuration and file validation
Some checks failed
Build and Publish Docker Image / deploy (push) Failing after 56s
Some checks failed
Build and Publish Docker Image / deploy (push) Failing after 56s
- Add `WARPBOX_RESUMABLE_CHUNK_MODE` and `WARPBOX_RESUMABLE_CHUNK_PATH` environment variables to configure temporary chunk storage. - Implement strict file validation for resuming uploads to ensure selected files match the pending session's metadata. - Add `PLANS.md` to document development stages, roadmap, and API specifications (including batching and resumable flows).
This commit is contained in:
@@ -12,6 +12,8 @@ WARPBOX_THUMBNAIL_EVERY=1m
|
|||||||
WARPBOX_RESUMABLE_UPLOADS_ENABLED=true
|
WARPBOX_RESUMABLE_UPLOADS_ENABLED=true
|
||||||
WARPBOX_RESUMABLE_CHUNK_MB=64
|
WARPBOX_RESUMABLE_CHUNK_MB=64
|
||||||
WARPBOX_RESUMABLE_RETENTION_HOURS=1
|
WARPBOX_RESUMABLE_RETENTION_HOURS=1
|
||||||
|
WARPBOX_RESUMABLE_CHUNK_MODE=same
|
||||||
|
WARPBOX_RESUMABLE_CHUNK_PATH=
|
||||||
WARPBOX_MAX_UPLOAD_SIZE_MB=16384
|
WARPBOX_MAX_UPLOAD_SIZE_MB=16384
|
||||||
WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true
|
WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true
|
||||||
WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512
|
WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512
|
||||||
|
|||||||
111
PLANS.md
Normal file
111
PLANS.md
Normal file
@@ -0,0 +1,111 @@
|
|||||||
|
# Warpbox Plans & Staged Development
|
||||||
|
|
||||||
|
This document captures the staged development history and roadmap for Warpbox. For day-to-day usage,
|
||||||
|
configuration, and deployment, see [README.md](./README.md).
|
||||||
|
|
||||||
|
## Stage 2 — Operator Tools
|
||||||
|
|
||||||
|
- `/admin/login` - token-based admin login.
|
||||||
|
- `/admin` - overview metrics: boxes, files, storage, recent uploads, protected/expired boxes.
|
||||||
|
- `/admin/files` - recent upload table with view and delete actions.
|
||||||
|
- Expired boxes and boxes that have reached their download limit are cleaned on startup and then
|
||||||
|
every `WARPBOX_CLEANUP_EVERY` when `WARPBOX_CLEANUP_ENABLED=true`.
|
||||||
|
- Missing image/video thumbnails are generated in a background worker every `WARPBOX_THUMBNAIL_EVERY`
|
||||||
|
when `WARPBOX_THUMBNAIL_ENABLED=true`.
|
||||||
|
|
||||||
|
## Stage 3 — Anonymous Integrations
|
||||||
|
|
||||||
|
Anonymous uploads return a private management link at creation time. Keep that link secret: anyone
|
||||||
|
with it can delete the entire upload box. The raw delete token is not stored and cannot be recovered
|
||||||
|
later.
|
||||||
|
|
||||||
|
Browser uploads still show `Open box` and `Copy URL` as the primary actions, with a smaller
|
||||||
|
`Manage or delete this upload` link in the completion panel.
|
||||||
|
|
||||||
|
Curl and custom uploaders can use the same endpoint:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Terminal-friendly output: one plain box URL.
|
||||||
|
curl -F file=@./report.pdf http://localhost:8080/api/v1/upload
|
||||||
|
|
||||||
|
# JSON output with boxUrl, thumbnailUrl, manageUrl, deleteUrl, zipUrl, and file entries.
|
||||||
|
curl -F sharex=@./screenshot.png \
|
||||||
|
-H 'Accept: application/json' \
|
||||||
|
http://localhost:8080/api/v1/upload
|
||||||
|
```
|
||||||
|
|
||||||
|
The upload endpoint accepts multipart fields named `file` and `sharex`. ShareX users can start from
|
||||||
|
`examples/sharex/warpbox-anonymous.sxcu`; update `RequestURL` to match your instance URL.
|
||||||
|
Authenticated uploads (your account's limits) add an `Authorization: Bearer <token>` header — mint a
|
||||||
|
token under **Account → Access tokens**. The JSON response uses ShareX placeholders `{json:boxUrl}`
|
||||||
|
(URL), `{json:thumbnailUrl}` (thumbnail), `{json:deleteUrl}` (deletion), and `{json:error}` (error
|
||||||
|
message).
|
||||||
|
|
||||||
|
### Grouping multiple files into one box (`X-Warpbox-Batch`)
|
||||||
|
|
||||||
|
By default every uploaded file becomes its own box. To put several files in a **single** box, send
|
||||||
|
the opt-in `X-Warpbox-Batch` header: requests that share the same header value (scoped per account,
|
||||||
|
or per IP for anonymous uploads) within 20s are appended to the same box. This lets a multi-file
|
||||||
|
ShareX selection — which ShareX sends as separate back-to-back requests — land as one shareable link.
|
||||||
|
The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file. Requests
|
||||||
|
without the header behave exactly as before.
|
||||||
|
|
||||||
|
### Resumable API flow
|
||||||
|
|
||||||
|
Custom clients can use the resumable JSON API for large files:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Create a resumable session from file metadata.
|
||||||
|
curl -s http://localhost:8080/api/v1/uploads/resumable \
|
||||||
|
-H 'Accept: application/json' \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{"files":[{"name":"large.bin","size":1048576,"contentType":"application/octet-stream"}],"expiresMinutes":1440}'
|
||||||
|
|
||||||
|
# 2. Upload exact-sized chunks using the returned sessionId, file id, and chunkSize.
|
||||||
|
dd if=./large.bin bs=8388608 count=1 skip=0 2>/dev/null | \
|
||||||
|
curl -X PUT --data-binary @- \
|
||||||
|
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
|
||||||
|
|
||||||
|
# 3. Complete the session after all chunks are present.
|
||||||
|
curl -X POST -H 'Accept: application/json' \
|
||||||
|
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/complete
|
||||||
|
```
|
||||||
|
|
||||||
|
The complete response is the same JSON shape as `POST /api/v1/upload`, including `boxUrl`,
|
||||||
|
`manageUrl`, `deleteUrl`, and file URLs. Send `Authorization: Bearer <token>` on every resumable
|
||||||
|
request to upload as an account.
|
||||||
|
|
||||||
|
Browser resumable uploads are configurable from `/admin/settings`: enabled/disabled, chunk size in
|
||||||
|
MB, draft retention hours, and chunk staging location. The default chunk mode stores temporary draft
|
||||||
|
chunks under `data/tmp/uploads/{session_id}`. A custom mode can point those chunks at another local
|
||||||
|
path, such as a mounted fast SSD. Chunk staging stays local even when the completed files are later
|
||||||
|
finalized into S3/SFTP/SMB/WebDAV storage. Completion returns the share link immediately; files may
|
||||||
|
show as `Processing` on the download page while the background finalizer streams them to the selected
|
||||||
|
storage backend. Cleanup removes expired uploading drafts but skips sessions already in processing.
|
||||||
|
|
||||||
|
## Stage 4 — Accounts + Personal Boxes
|
||||||
|
|
||||||
|
- `/register` bootstraps the first admin account only when no users exist.
|
||||||
|
- `/login` and `/logout` provide cookie-based web sessions.
|
||||||
|
- `/app` is the personal dashboard for logged-in users, showing owned boxes, storage usage, upload
|
||||||
|
history, and flat collections. Uploading still happens from the homepage.
|
||||||
|
- `/admin/users` lets admins create invite links, disable/reactivate users, and generate reset links.
|
||||||
|
- Logged-in browser uploads from `/` still use `POST /api/v1/upload`, but the resulting box is stored
|
||||||
|
with owner and optional collection metadata.
|
||||||
|
- Admin users are exempt from the global max upload size on the homepage upload flow. Future per-user
|
||||||
|
quotas should apply to this same upload path rather than creating a second uploader.
|
||||||
|
- `/admin/settings` controls anonymous uploads, anonymous max upload size, daily upload caps, default
|
||||||
|
user storage quota, and usage retention.
|
||||||
|
- `/admin/users` shows storage/daily usage and lets admins set per-user storage quota overrides.
|
||||||
|
- `/admin/storage` manages the built-in local file backend and S3-compatible bucket backends.
|
||||||
|
- `/admin/bans` manages manual IP/CIDR bans and optional automatic bans for suspicious probes and
|
||||||
|
repeated login failures. Auto-ban is off by default and configured from the admin UI.
|
||||||
|
- Upload limits now include daily bytes, daily box counts, active box counts, short-window request
|
||||||
|
limits, max expiration days, local storage capacity in GB, and per-user policy overrides.
|
||||||
|
- Uploaded file content, thumbnails, and private box metadata use the selected storage backend. The
|
||||||
|
bbolt database and JSON logs remain local under `./data/db` and `./data/logs`.
|
||||||
|
- Anonymous uploads, ShareX uploads, unlisted public box links, password protection, expiry, delete
|
||||||
|
tokens, thumbnails, and cleanup continue to work as before.
|
||||||
|
|
||||||
|
Email delivery is intentionally deferred. Invite and reset links are copyable today; future SMTP
|
||||||
|
support will power public forgot-password and optional email delivery.
|
||||||
346
README.md
346
README.md
@@ -1,6 +1,20 @@
|
|||||||
# Warpbox.dev
|
# Warpbox.dev
|
||||||
|
|
||||||
This repository contains the Go backend base for `warpbox.dev`, a self-hosted transfer-first file sharing application.
|
Warpbox is a self-hosted, transfer first file sharing application written in Go. It renders
|
||||||
|
server side templates and serves static assets directly.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- Anonymous and authenticated uploads from the browser, `curl`, or ShareX.
|
||||||
|
- Warpbox-native resumable uploads with a JSON API for large files.
|
||||||
|
- Upload boxes with expiry, optional download limits, password protection, and one time delete tokens.
|
||||||
|
- User accounts with personal dashboards, collections, storage quotas, and invite based registration.
|
||||||
|
- Admin tooling for metrics, file management, storage backends, upload policy, and IP bans.
|
||||||
|
- Local and S3 compatible storage backends.
|
||||||
|
- Background jobs for cleanup and thumbnail generation.
|
||||||
|
- Emoji reaction packs loaded from the runtime data directory.
|
||||||
|
|
||||||
|
> Looking for the roadmap and the staged development history? See [PLANS.md](./PLANS.md).
|
||||||
|
|
||||||
## Run
|
## Run
|
||||||
|
|
||||||
@@ -10,11 +24,25 @@ This repository contains the Go backend base for `warpbox.dev`, a self-hosted tr
|
|||||||
|
|
||||||
The default server listens on `:8080`.
|
The default server listens on `:8080`.
|
||||||
|
|
||||||
Upload size limits are configured in megabytes through `WARPBOX_MAX_UPLOAD_SIZE_MB`.
|
For one off Go commands, run them from the backend module:
|
||||||
Fractions are supported, so `0.5Mb` is 512 KiB and `1.5Mb` is 1536 KiB.
|
|
||||||
|
|
||||||
Upload policy defaults are also configured in megabytes and can later be changed from
|
```bash
|
||||||
`/admin/settings`:
|
cd backend
|
||||||
|
go run ./cmd/warpbox
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
All configuration comes from environment variables. The dev script sources `scripts/env/dev.env`.
|
||||||
|
|
||||||
|
### Upload size
|
||||||
|
|
||||||
|
Upload size limits are configured in megabytes through `WARPBOX_MAX_UPLOAD_SIZE_MB`. Fractions are
|
||||||
|
supported, so `0.5Mb` is 512 KiB and `1.5Mb` is 1536 KiB.
|
||||||
|
|
||||||
|
### Upload policy defaults
|
||||||
|
|
||||||
|
These defaults can later be changed from `/admin/settings`:
|
||||||
|
|
||||||
- `WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true`
|
- `WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true`
|
||||||
- `WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512`
|
- `WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512`
|
||||||
@@ -36,36 +64,147 @@ Upload policy defaults are also configured in megabytes and can later be changed
|
|||||||
- `WARPBOX_RESUMABLE_UPLOADS_ENABLED=true`
|
- `WARPBOX_RESUMABLE_UPLOADS_ENABLED=true`
|
||||||
- `WARPBOX_RESUMABLE_CHUNK_MB=8`
|
- `WARPBOX_RESUMABLE_CHUNK_MB=8`
|
||||||
- `WARPBOX_RESUMABLE_RETENTION_HOURS=24`
|
- `WARPBOX_RESUMABLE_RETENTION_HOURS=24`
|
||||||
- `WARPBOX_TRUSTED_PROXIES=` controls whether forwarded client IP headers are accepted only from specific proxy IPs/CIDRs. See [SECURITY_PROXY.md](./SECURITY_PROXY.md).
|
- `WARPBOX_RESUMABLE_CHUNK_MODE=same`
|
||||||
|
- `WARPBOX_RESUMABLE_CHUNK_PATH=`
|
||||||
|
- `WARPBOX_TRUSTED_PROXIES=` controls whether forwarded client IP headers are accepted only from
|
||||||
|
specific proxy IPs/CIDRs. See [SECURITY_PROXY.md](./SECURITY_PROXY.md).
|
||||||
|
|
||||||
|
Resumable settings are seeded from the environment and can then be edited from `/admin/settings`.
|
||||||
|
Saved admin settings override these env defaults. `WARPBOX_RESUMABLE_CHUNK_MODE=same` stores draft
|
||||||
|
chunks in the normal local temp path, `data/tmp/uploads/{session_id}` under `WARPBOX_DATA_DIR`.
|
||||||
|
`WARPBOX_RESUMABLE_CHUNK_MODE=custom` uses `WARPBOX_RESUMABLE_CHUNK_PATH` instead, for example a
|
||||||
|
mounted fast SSD path. Chunk storage is always local temporary staging; completed files are finalized
|
||||||
|
into the selected storage backend after all chunks arrive.
|
||||||
|
|
||||||
|
### Timeouts
|
||||||
|
|
||||||
|
Large uploads are expected to take minutes on normal home/server connections. Keep
|
||||||
|
`WARPBOX_READ_TIMEOUT=0s` and `WARPBOX_WRITE_TIMEOUT=0s` so Go does not close the connection
|
||||||
|
mid upload; `WARPBOX_READ_HEADER_TIMEOUT=15s` still protects header reads from slowloris style
|
||||||
|
connections.
|
||||||
|
|
||||||
|
### Data directory
|
||||||
|
|
||||||
Runtime data is configured with `WARPBOX_DATA_DIR` and defaults to `./data` in the dev environment.
|
Runtime data is configured with `WARPBOX_DATA_DIR` and defaults to `./data` in the dev environment.
|
||||||
The dev script resolves that path from the repository root.
|
The dev script resolves that path from the repository root.
|
||||||
|
|
||||||
Large uploads are expected to take minutes on normal home/server connections. Keep
|
### Background jobs
|
||||||
`WARPBOX_READ_TIMEOUT=0s` and `WARPBOX_WRITE_TIMEOUT=0s` so Go does not close the connection
|
|
||||||
mid-upload; `WARPBOX_READ_HEADER_TIMEOUT=15s` still protects header reads from slowloris-style
|
|
||||||
connections.
|
|
||||||
|
|
||||||
Browser uploads use Warpbox-native resumable uploads by default. Chunks are stored temporarily under
|
|
||||||
`data/tmp/uploads/{session_id}` and then streamed into the selected storage backend when the upload
|
|
||||||
is completed. Stale sessions are cleaned by the cleanup job after `WARPBOX_RESUMABLE_RETENTION_HOURS`.
|
|
||||||
|
|
||||||
Background jobs are enabled with `WARPBOX_JOBS_ENABLED=true`. Individual jobs can be toggled with
|
Background jobs are enabled with `WARPBOX_JOBS_ENABLED=true`. Individual jobs can be toggled with
|
||||||
`WARPBOX_CLEANUP_ENABLED` and `WARPBOX_THUMBNAIL_ENABLED`, and their schedules are configured with
|
`WARPBOX_CLEANUP_ENABLED` and `WARPBOX_THUMBNAIL_ENABLED`, and their schedules are configured with
|
||||||
`WARPBOX_CLEANUP_EVERY` and `WARPBOX_THUMBNAIL_EVERY`.
|
`WARPBOX_CLEANUP_EVERY` and `WARPBOX_THUMBNAIL_EVERY`.
|
||||||
|
|
||||||
|
- **Cleanup**: expired boxes and boxes that have reached their download limit are cleaned on startup
|
||||||
|
and then on schedule. Stale resumable sessions are removed after `WARPBOX_RESUMABLE_RETENTION_HOURS`.
|
||||||
|
- **Thumbnails**: missing image/video thumbnails are generated in a background worker.
|
||||||
|
|
||||||
|
## First run bootstrap
|
||||||
|
|
||||||
On a fresh data directory, visit `/register` to create the first account. That first user becomes
|
On a fresh data directory, visit `/register` to create the first account. That first user becomes
|
||||||
the instance admin and normal registration closes after bootstrap. Admins can create copyable invite
|
the instance admin and normal registration closes after bootstrap. Admins can create copyable invite
|
||||||
links from `/admin/users`.
|
links from `/admin/users`.
|
||||||
|
|
||||||
The env admin token still exists as emergency fallback access. Set `WARPBOX_ADMIN_TOKEN` and use it
|
The env admin token exists as emergency fallback access. Set `WARPBOX_ADMIN_TOKEN` and use it at
|
||||||
at `/admin/login` if you need to recover access without a user session.
|
`/admin/login` if you need to recover access without a user session.
|
||||||
|
|
||||||
|
## Uploads
|
||||||
|
|
||||||
|
Browser uploads use Warpbox native resumable uploads by default. Resumable behavior is configurable
|
||||||
|
from `/admin/settings`, including enable/disable, chunk size, retention, and whether chunks use the
|
||||||
|
default local temp path or a custom local path such as a fast SSD. When all chunks arrive, Warpbox
|
||||||
|
returns the share link immediately and marks files as `Processing` until the background finalizer
|
||||||
|
streams them into the selected storage backend. Draft chunks are deleted once finalization succeeds.
|
||||||
|
Expired uploading drafts are cleaned after the configured retention window; sessions already in
|
||||||
|
`Processing` are protected from cleanup while finalization is running.
|
||||||
|
|
||||||
|
### Anonymous uploads
|
||||||
|
|
||||||
|
Anonymous uploads return a private management link at creation time. Keep that link secret: anyone
|
||||||
|
with it can delete the entire upload box. The raw delete token is not stored and cannot be recovered
|
||||||
|
later. Browser uploads show `Open box` and `Copy URL` as the primary actions, with a smaller
|
||||||
|
`Manage or delete this upload` link in the completion panel.
|
||||||
|
|
||||||
|
### `curl` and ShareX
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Terminal-friendly output: one plain box URL.
|
||||||
|
curl -F file=@./report.pdf http://localhost:8080/api/v1/upload
|
||||||
|
|
||||||
|
# JSON output with boxUrl, thumbnailUrl, manageUrl, deleteUrl, zipUrl, and file entries.
|
||||||
|
curl -F sharex=@./screenshot.png \
|
||||||
|
-H 'Accept: application/json' \
|
||||||
|
http://localhost:8080/api/v1/upload
|
||||||
|
```
|
||||||
|
|
||||||
|
The upload endpoint accepts multipart fields named `file` and `sharex`. ShareX users can start from
|
||||||
|
`examples/sharex/warpbox-anonymous.sxcu`; update `RequestURL` to match your instance URL.
|
||||||
|
Authenticated uploads (your account's limits) add an `Authorization: Bearer <token>` header mint a
|
||||||
|
token under **Account → Access tokens**. The JSON response uses ShareX placeholders `{json:boxUrl}`
|
||||||
|
(URL), `{json:thumbnailUrl}` (thumbnail), `{json:deleteUrl}` (deletion), and `{json:error}` (error
|
||||||
|
message).
|
||||||
|
|
||||||
|
### Grouping multiple files into one box (`X-Warpbox-Batch`)
|
||||||
|
|
||||||
|
By default every uploaded file becomes its own box. To put several files in a **single** box, send
|
||||||
|
the opt-in `X-Warpbox-Batch` header: requests that share the same header value (scoped per account,
|
||||||
|
or per IP for anonymous uploads) within 20s are appended to the same box. This lets a multi-file
|
||||||
|
ShareX selection which ShareX sends as separate back-to-back requests land as one shareable link.
|
||||||
|
The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file. Requests
|
||||||
|
without the header behave exactly as before.
|
||||||
|
|
||||||
|
### Resumable API flow
|
||||||
|
|
||||||
|
Custom clients can use the resumable JSON API for large files:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Create a resumable session from file metadata.
|
||||||
|
curl -s http://localhost:8080/api/v1/uploads/resumable \
|
||||||
|
-H 'Accept: application/json' \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{"files":[{"name":"large.bin","size":1048576,"contentType":"application/octet-stream"}],"expiresMinutes":1440}'
|
||||||
|
|
||||||
|
# 2. Upload exact-sized chunks using the returned sessionId, file id, and chunkSize.
|
||||||
|
dd if=./large.bin bs=8388608 count=1 skip=0 2>/dev/null | \
|
||||||
|
curl -X PUT --data-binary @- \
|
||||||
|
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
|
||||||
|
|
||||||
|
# 3. Complete the session after all chunks are present.
|
||||||
|
curl -X POST -H 'Accept: application/json' \
|
||||||
|
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/complete
|
||||||
|
```
|
||||||
|
|
||||||
|
The complete response is the same JSON shape as `POST /api/v1/upload`, including `boxUrl`,
|
||||||
|
`manageUrl`, `deleteUrl`, and file URLs. Send `Authorization: Bearer <token>` on every resumable
|
||||||
|
request to upload as an account.
|
||||||
|
|
||||||
|
## Accounts and admin
|
||||||
|
|
||||||
|
- `/register` bootstraps the first admin account only when no users exist.
|
||||||
|
- `/login` and `/logout` provide cookie-based web sessions.
|
||||||
|
- `/app` is the personal dashboard for logged-in users, showing owned boxes, storage usage, upload
|
||||||
|
history, and flat collections. Uploading still happens from the homepage.
|
||||||
|
- `/admin` shows overview metrics: boxes, files, storage, recent uploads, protected/expired boxes.
|
||||||
|
- `/admin/files` is a recent upload table with view and delete actions.
|
||||||
|
- `/admin/users` lets admins create invite links, disable/reactivate users, generate reset links,
|
||||||
|
view storage/daily usage, and set per-user storage quota overrides.
|
||||||
|
- `/admin/settings` controls anonymous uploads, anonymous max upload size, daily upload caps, default
|
||||||
|
user storage quota, and usage retention.
|
||||||
|
- `/admin/storage` manages the built-in local file backend and S3-compatible bucket backends.
|
||||||
|
- `/admin/bans` manages manual IP/CIDR bans and optional automatic bans for suspicious probes and
|
||||||
|
repeated login failures. Auto-ban is off by default and configured from the admin UI.
|
||||||
|
|
||||||
|
Logged-in browser uploads from `/` use `POST /api/v1/upload`, and the resulting box is stored with
|
||||||
|
owner and optional collection metadata. Admin users are exempt from the global max upload size on the
|
||||||
|
homepage upload flow.
|
||||||
|
|
||||||
|
Email delivery is intentionally deferred. Invite and reset links are copyable today; future SMTP
|
||||||
|
support will power public forgot-password and optional email delivery.
|
||||||
|
|
||||||
## Emoji reaction packs
|
## Emoji reaction packs
|
||||||
|
|
||||||
File reactions use emoji packs from the runtime data directory, not from the application code. By
|
File reactions use emoji packs from the runtime data directory, not from the application code. By
|
||||||
default that means `./data/emoji`; if you change `WARPBOX_DATA_DIR`, use
|
default that means `./data/emoji`; if you change `WARPBOX_DATA_DIR`, use `$WARPBOX_DATA_DIR/emoji`
|
||||||
`$WARPBOX_DATA_DIR/emoji` instead.
|
instead.
|
||||||
|
|
||||||
Each folder under `./data/emoji` becomes one emoji tab in the reaction picker. Put image files
|
Each folder under `./data/emoji` becomes one emoji tab in the reaction picker. Put image files
|
||||||
directly inside the pack folder:
|
directly inside the pack folder:
|
||||||
@@ -89,22 +228,17 @@ data/
|
|||||||
└── shipped.png
|
└── shipped.png
|
||||||
```
|
```
|
||||||
|
|
||||||
In this example, the picker shows tabs named `Openmoji`, `Pixel pack`, and `Custom work`.
|
In this example, the picker shows tabs named `Openmoji`, `Pixel pack`, and `Custom work`. Supported
|
||||||
Supported emoji image extensions are `.svg`, `.webp`, `.png`, `.jpg`, `.jpeg`, and `.gif`.
|
emoji image extensions are `.svg`, `.webp`, `.png`, `.jpg`, `.jpeg`, and `.gif`.
|
||||||
|
|
||||||
For one-off Go commands, run them from the backend module:
|
## Deployment
|
||||||
|
|
||||||
```bash
|
### Docker / Podman
|
||||||
cd backend
|
|
||||||
go run ./cmd/warpbox
|
|
||||||
```
|
|
||||||
|
|
||||||
## Docker / Podman
|
|
||||||
|
|
||||||
Copy the example environment file and adjust values such as `WARPBOX_BASE_URL` and
|
Copy the example environment file and adjust values such as `WARPBOX_BASE_URL` and
|
||||||
`WARPBOX_ADMIN_TOKEN` before running the container:
|
`WARPBOX_ADMIN_TOKEN` before running the container. Copy the example
|
||||||
|
[docker-compose.example.yml](./docker-compose.example.yml) to
|
||||||
Copy the example [docker-compose.example.yml](./docker-compose.example.yml) to [docker-compose.yml](./docker-compose.yml), modify as need-be
|
[docker-compose.yml](./docker-compose.yml), modify as need-be:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cp .env.example .env
|
cp .env.example .env
|
||||||
@@ -112,20 +246,19 @@ docker compose -f docker-compose.yml up --build
|
|||||||
```
|
```
|
||||||
|
|
||||||
The compose example also works with Podman compatible compose tools. Its data volume uses
|
The compose example also works with Podman compatible compose tools. Its data volume uses
|
||||||
`./data:/data:Z` for SELinux relabeling, and the container overrides runtime paths to use
|
`./data:/data:Z` for SELinux relabeling, and the container overrides runtime paths to use `/data`,
|
||||||
`/data`, `/app/static`, and `/app/templates`.
|
`/app/static`, and `/app/templates`. The image exposes the health endpoint `/health`, which Docker
|
||||||
|
and compose healthchecks use.
|
||||||
|
|
||||||
The image exposes the health endpoint: `/health`. Docker and compose healthchecks use it.
|
### Reverse proxy security
|
||||||
|
|
||||||
## Reverse Proxy Security
|
|
||||||
|
|
||||||
Warpbox uses the resolved client IP for anonymous limits, manual bans, and automatic bans. The
|
Warpbox uses the resolved client IP for anonymous limits, manual bans, and automatic bans. The
|
||||||
default behavior trusts `X-Forwarded-For` and `X-Real-IP` so a normal Caddy reverse proxy works
|
default behavior trusts `X-Forwarded-For` and `X-Real-IP` so a normal Caddy reverse proxy works
|
||||||
without extra setup. For hardened deployments where the app port might be reachable from more than
|
without extra setup. For hardened deployments where the app port might be reachable from more than one
|
||||||
one network, set `WARPBOX_TRUSTED_PROXIES` to trusted proxy IPs/CIDRs. See
|
network, set `WARPBOX_TRUSTED_PROXIES` to trusted proxy IPs/CIDRs. See
|
||||||
[SECURITY_PROXY.md](./SECURITY_PROXY.md) for Caddy examples and Docker/systemd notes.
|
[SECURITY_PROXY.md](./SECURITY_PROXY.md) for Caddy examples and Docker/systemd notes.
|
||||||
|
|
||||||
## Systemd
|
### Systemd
|
||||||
|
|
||||||
Build the binary on the server, create a dedicated user, and keep runtime data outside the repo:
|
Build the binary on the server, create a dedicated user, and keep runtime data outside the repo:
|
||||||
|
|
||||||
@@ -187,7 +320,23 @@ sudo systemctl status warpbox
|
|||||||
|
|
||||||
Put Caddy in front of `127.0.0.1:6070` and keep the Warpbox port closed to the public internet.
|
Put Caddy in front of `127.0.0.1:6070` and keep the Warpbox port closed to the public internet.
|
||||||
|
|
||||||
## Layout
|
## Runtime data
|
||||||
|
|
||||||
|
Warpbox keeps local runtime data under the configured data directory:
|
||||||
|
|
||||||
|
- `data/files/{box_id}/@each@{file_id}.ext` - uploaded file contents when the local backend is selected.
|
||||||
|
- `data/files/{box_id}/@thumb@{file_id}.jpg` - generated previews when the local backend is selected.
|
||||||
|
- `data/tmp/uploads/{session_id}` - temporary local chunks for unfinished resumable uploads when
|
||||||
|
the default chunk mode is selected.
|
||||||
|
- `data/db/warpbox.bbolt` - bbolt metadata database for boxes, file records, users, sessions,
|
||||||
|
invites, collections, upload policy settings, daily usage records, manual bans, automatic ban
|
||||||
|
settings, abuse counters, and malicious path rules.
|
||||||
|
- `data/logs/{YYYY-MM-DD}.log` - JSONL logs, one event per line.
|
||||||
|
|
||||||
|
Uploaded file content, thumbnails, and private box metadata use the selected storage backend. The
|
||||||
|
bbolt database and JSON logs always remain local under `./data/db` and `./data/logs`.
|
||||||
|
|
||||||
|
## Project layout
|
||||||
|
|
||||||
- `backend/cmd/warpbox` - main application entry point.
|
- `backend/cmd/warpbox` - main application entry point.
|
||||||
- `backend/libs/config` - environment-backed configuration.
|
- `backend/libs/config` - environment-backed configuration.
|
||||||
@@ -195,7 +344,7 @@ Put Caddy in front of `127.0.0.1:6070` and keep the Warpbox port closed to the p
|
|||||||
- `backend/libs/handlers` - HTTP handlers for pages, API, health, static files.
|
- `backend/libs/handlers` - HTTP handlers for pages, API, health, static files.
|
||||||
- `backend/libs/jobs` - background job registration and job loop definitions.
|
- `backend/libs/jobs` - background job registration and job loop definitions.
|
||||||
- `backend/libs/middleware` - request logging, recovery, security headers, gzip, request IDs.
|
- `backend/libs/middleware` - request logging, recovery, security headers, gzip, request IDs.
|
||||||
- `backend/libs/services` - business logic boundaries, starting with upload limits.
|
- `backend/libs/services` - business logic boundaries.
|
||||||
- `backend/libs/helpers` - small reusable helpers.
|
- `backend/libs/helpers` - small reusable helpers.
|
||||||
- `backend/libs/web` - Go template renderer.
|
- `backend/libs/web` - Go template renderer.
|
||||||
- `backend/templates` - server-rendered Go templates.
|
- `backend/templates` - server-rendered Go templates.
|
||||||
@@ -204,118 +353,7 @@ Put Caddy in front of `127.0.0.1:6070` and keep the Warpbox port closed to the p
|
|||||||
- `scripts/env/dev.env.example` - tracked development environment template.
|
- `scripts/env/dev.env.example` - tracked development environment template.
|
||||||
- `scripts/env/dev.env` - local development environment, ignored by git.
|
- `scripts/env/dev.env` - local development environment, ignored by git.
|
||||||
|
|
||||||
## Stage 2 Operator Tools
|
## Static asset policy
|
||||||
|
|
||||||
- `/admin/login` - token-based admin login.
|
The static handler sets long-lived immutable caching for images, video, audio, and fonts, shorter
|
||||||
- `/admin` - overview metrics: boxes, files, storage, recent uploads, protected/expired boxes.
|
caching for CSS/JS, and gzip compression for compressible responses.
|
||||||
- `/admin/files` - recent upload table with view and delete actions.
|
|
||||||
- Expired boxes and boxes that have reached their download limit are cleaned on startup and then every `WARPBOX_CLEANUP_EVERY` when `WARPBOX_CLEANUP_ENABLED=true`.
|
|
||||||
- Missing image/video thumbnails are generated in a background worker every `WARPBOX_THUMBNAIL_EVERY` when `WARPBOX_THUMBNAIL_ENABLED=true`.
|
|
||||||
|
|
||||||
## Stage 3 Anonymous Integrations
|
|
||||||
|
|
||||||
Anonymous uploads now return a private management link at creation time. Keep that link secret:
|
|
||||||
anyone with it can delete the entire upload box. The raw delete token is not stored and cannot be
|
|
||||||
recovered later.
|
|
||||||
|
|
||||||
Browser uploads still show `Open box` and `Copy URL` as the primary actions, with a smaller
|
|
||||||
`Manage or delete this upload` link in the completion panel.
|
|
||||||
|
|
||||||
Curl and custom uploaders can use the same endpoint:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Terminal-friendly output: one plain box URL.
|
|
||||||
curl -F file=@./report.pdf http://localhost:8080/api/v1/upload
|
|
||||||
|
|
||||||
# JSON output with boxUrl, thumbnailUrl, manageUrl, deleteUrl, zipUrl, and file entries.
|
|
||||||
curl -F sharex=@./screenshot.png \
|
|
||||||
-H 'Accept: application/json' \
|
|
||||||
http://localhost:8080/api/v1/upload
|
|
||||||
```
|
|
||||||
|
|
||||||
The upload endpoint accepts multipart fields named `file` and `sharex`. ShareX users can start
|
|
||||||
from `examples/sharex/warpbox-anonymous.sxcu`; update `RequestURL` to match your instance URL.
|
|
||||||
Authenticated uploads (your account's limits) add an `Authorization: Bearer <token>` header — mint
|
|
||||||
a token under **Account → Access tokens**. The JSON response uses ShareX placeholders
|
|
||||||
`{json:boxUrl}` (URL), `{json:thumbnailUrl}` (thumbnail), `{json:deleteUrl}` (deletion), and
|
|
||||||
`{json:error}` (error message).
|
|
||||||
|
|
||||||
### Grouping multiple files into one box (`X-Warpbox-Batch`)
|
|
||||||
|
|
||||||
By default every uploaded file becomes its own box. To put several files in a **single** box, send
|
|
||||||
the opt-in `X-Warpbox-Batch` header: requests that share the same header value (scoped per account,
|
|
||||||
or per IP for anonymous uploads) within 20s are appended to the same box. This lets a multi-file
|
|
||||||
ShareX selection — which ShareX sends as separate back-to-back requests — land as one shareable
|
|
||||||
link. The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file.
|
|
||||||
Requests without the header behave exactly as before.
|
|
||||||
|
|
||||||
### Resumable API flow
|
|
||||||
|
|
||||||
Custom clients can use the resumable JSON API for large files:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 1. Create a resumable session from file metadata.
|
|
||||||
curl -s http://localhost:8080/api/v1/uploads/resumable \
|
|
||||||
-H 'Accept: application/json' \
|
|
||||||
-H 'Content-Type: application/json' \
|
|
||||||
-d '{"files":[{"name":"large.bin","size":1048576,"contentType":"application/octet-stream"}],"expiresMinutes":1440}'
|
|
||||||
|
|
||||||
# 2. Upload exact-sized chunks using the returned sessionId, file id, and chunkSize.
|
|
||||||
dd if=./large.bin bs=8388608 count=1 skip=0 2>/dev/null | \
|
|
||||||
curl -X PUT --data-binary @- \
|
|
||||||
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
|
|
||||||
|
|
||||||
# 3. Complete the session after all chunks are present.
|
|
||||||
curl -X POST -H 'Accept: application/json' \
|
|
||||||
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/complete
|
|
||||||
```
|
|
||||||
|
|
||||||
The complete response is the same JSON shape as `POST /api/v1/upload`, including `boxUrl`,
|
|
||||||
`manageUrl`, `deleteUrl`, and file URLs. Send `Authorization: Bearer <token>` on every resumable
|
|
||||||
request to upload as an account.
|
|
||||||
|
|
||||||
## Stage 4 Accounts + Personal Boxes
|
|
||||||
|
|
||||||
- `/register` bootstraps the first admin account only when no users exist.
|
|
||||||
- `/login` and `/logout` provide cookie-based web sessions.
|
|
||||||
- `/app` is the personal dashboard for logged-in users, showing owned boxes, storage usage, upload
|
|
||||||
history, and flat collections. Uploading still happens from the homepage.
|
|
||||||
- `/admin/users` lets admins create invite links, disable/reactivate users, and generate reset links.
|
|
||||||
- Logged-in browser uploads from `/` still use `POST /api/v1/upload`, but the resulting box is
|
|
||||||
stored with owner and optional collection metadata.
|
|
||||||
- Admin users are exempt from the global max upload size on the homepage upload flow. Future
|
|
||||||
per-user quotas should apply to this same upload path rather than creating a second uploader.
|
|
||||||
- `/admin/settings` controls anonymous uploads, anonymous max upload size, daily upload caps, default
|
|
||||||
user storage quota, and usage retention.
|
|
||||||
- `/admin/users` shows storage/daily usage and lets admins set per-user storage quota overrides.
|
|
||||||
- `/admin/storage` manages the built-in local file backend and S3-compatible bucket backends.
|
|
||||||
- `/admin/bans` manages manual IP/CIDR bans and optional automatic bans for suspicious probes and
|
|
||||||
repeated login failures. Auto-ban is off by default and configured from the admin UI.
|
|
||||||
- Upload limits now include daily bytes, daily box counts, active box counts, short-window request
|
|
||||||
limits, max expiration days, local storage capacity in GB, and per-user policy overrides.
|
|
||||||
- Uploaded file content, thumbnails, and private box metadata use the selected storage backend.
|
|
||||||
The bbolt database and JSON logs remain local under `./data/db` and `./data/logs`.
|
|
||||||
- Anonymous uploads, ShareX uploads, unlisted public box links, password protection, expiry, delete
|
|
||||||
tokens, thumbnails, and cleanup continue to work as before.
|
|
||||||
|
|
||||||
Email delivery is intentionally deferred. Invite and reset links are copyable today; future SMTP
|
|
||||||
support will power public forgot-password and optional email delivery.
|
|
||||||
|
|
||||||
## Runtime Data
|
|
||||||
|
|
||||||
Warpbox keeps local runtime data under the configured data directory:
|
|
||||||
|
|
||||||
- `data/files/{box_id}/@each@{file_id}.ext` - uploaded file contents when the local backend is selected.
|
|
||||||
- `data/files/{box_id}/@thumb@{file_id}.jpg` - generated previews when the local backend is selected.
|
|
||||||
- `data/tmp/uploads/{session_id}` - temporary chunks for unfinished resumable uploads.
|
|
||||||
- `data/db/warpbox.bbolt` - bbolt metadata database for boxes and file records.
|
|
||||||
- `data/db/warpbox.bbolt` also stores users, sessions, invites, and collections.
|
|
||||||
- `data/db/warpbox.bbolt` stores upload policy settings and daily usage records keyed by plain IP
|
|
||||||
for anonymous uploads and user ID for signed-in uploads.
|
|
||||||
- `data/db/warpbox.bbolt` stores manual bans, automatic ban settings, abuse counters, and malicious
|
|
||||||
path rules.
|
|
||||||
- `data/logs/{YYYY-MM-DD}.log` - JSONL logs, one event per line.
|
|
||||||
|
|
||||||
## Static Asset Policy
|
|
||||||
|
|
||||||
The static handler sets long-lived immutable caching for images, video, audio, and fonts, shorter caching for CSS/JS, and gzip compression for compressible responses.
|
|
||||||
|
|||||||
@@ -55,6 +55,11 @@ type SettingsDefaults struct {
|
|||||||
ShortWindowSeconds int
|
ShortWindowSeconds int
|
||||||
AnonymousStorageBackend string
|
AnonymousStorageBackend string
|
||||||
UserStorageBackend string
|
UserStorageBackend string
|
||||||
|
ResumableUploadsEnabled bool
|
||||||
|
ResumableChunkSizeMB float64
|
||||||
|
ResumableRetentionHours int
|
||||||
|
ResumableChunkMode string
|
||||||
|
ResumableChunkPath string
|
||||||
}
|
}
|
||||||
|
|
||||||
func Load() (Config, error) {
|
func Load() (Config, error) {
|
||||||
@@ -100,8 +105,13 @@ func Load() (Config, error) {
|
|||||||
ShortWindowSeconds: envInt("WARPBOX_SHORT_WINDOW_SECONDS", 60),
|
ShortWindowSeconds: envInt("WARPBOX_SHORT_WINDOW_SECONDS", 60),
|
||||||
AnonymousStorageBackend: envString("WARPBOX_ANONYMOUS_STORAGE_BACKEND", "local"),
|
AnonymousStorageBackend: envString("WARPBOX_ANONYMOUS_STORAGE_BACKEND", "local"),
|
||||||
UserStorageBackend: envString("WARPBOX_USER_STORAGE_BACKEND", "local"),
|
UserStorageBackend: envString("WARPBOX_USER_STORAGE_BACKEND", "local"),
|
||||||
|
ResumableChunkMode: envString("WARPBOX_RESUMABLE_CHUNK_MODE", "same"),
|
||||||
|
ResumableChunkPath: envString("WARPBOX_RESUMABLE_CHUNK_PATH", ""),
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
cfg.DefaultSettings.ResumableUploadsEnabled = cfg.ResumableUploadsEnabled
|
||||||
|
cfg.DefaultSettings.ResumableChunkSizeMB = float64(cfg.ResumableChunkSize) / 1024 / 1024
|
||||||
|
cfg.DefaultSettings.ResumableRetentionHours = int(cfg.ResumableRetention / time.Hour)
|
||||||
|
|
||||||
if cfg.BaseURL == "" {
|
if cfg.BaseURL == "" {
|
||||||
return Config{}, fmt.Errorf("WARPBOX_BASE_URL cannot be empty")
|
return Config{}, fmt.Errorf("WARPBOX_BASE_URL cannot be empty")
|
||||||
|
|||||||
@@ -574,6 +574,7 @@ func (a *App) AdminSettingsPost(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
settings.AnonymousUploadsEnabled = r.FormValue("anonymous_uploads_enabled") == "on"
|
settings.AnonymousUploadsEnabled = r.FormValue("anonymous_uploads_enabled") == "on"
|
||||||
|
settings.ResumableUploadsEnabled = r.FormValue("resumable_uploads_enabled") == "on"
|
||||||
if value := parsePositiveInt(r.FormValue("usage_retention_days")); value > 0 {
|
if value := parsePositiveInt(r.FormValue("usage_retention_days")); value > 0 {
|
||||||
settings.UsageRetentionDays = value
|
settings.UsageRetentionDays = value
|
||||||
}
|
}
|
||||||
@@ -604,6 +605,16 @@ func (a *App) AdminSettingsPost(w http.ResponseWriter, r *http.Request) {
|
|||||||
if value := parsePositiveInt(r.FormValue("short_window_seconds")); value > 0 {
|
if value := parsePositiveInt(r.FormValue("short_window_seconds")); value > 0 {
|
||||||
settings.ShortWindowSeconds = value
|
settings.ShortWindowSeconds = value
|
||||||
}
|
}
|
||||||
|
if value := parsePositiveFloat(r.FormValue("resumable_chunk_size_mb")); value > 0 {
|
||||||
|
settings.ResumableChunkSizeMB = value
|
||||||
|
}
|
||||||
|
if value := parsePositiveInt(r.FormValue("resumable_retention_hours")); value > 0 {
|
||||||
|
settings.ResumableRetentionHours = value
|
||||||
|
}
|
||||||
|
if value := strings.TrimSpace(r.FormValue("resumable_chunk_mode")); value != "" {
|
||||||
|
settings.ResumableChunkMode = value
|
||||||
|
}
|
||||||
|
settings.ResumableChunkPath = strings.TrimSpace(r.FormValue("resumable_chunk_path"))
|
||||||
if value := r.FormValue("anonymous_storage_backend"); value != "" {
|
if value := r.FormValue("anonymous_storage_backend"); value != "" {
|
||||||
settings.AnonymousStorageBackend = value
|
settings.AnonymousStorageBackend = value
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -146,6 +146,7 @@ func (a *App) RegisterRoutes(mux *http.ServeMux) {
|
|||||||
mux.HandleFunc("POST /api/v1/uploads/resumable/{sessionID}/files", a.AddResumableFiles)
|
mux.HandleFunc("POST /api/v1/uploads/resumable/{sessionID}/files", a.AddResumableFiles)
|
||||||
mux.HandleFunc("PUT /api/v1/uploads/resumable/{sessionID}/files/{fileID}/chunks/{index}", a.PutResumableChunk)
|
mux.HandleFunc("PUT /api/v1/uploads/resumable/{sessionID}/files/{fileID}/chunks/{index}", a.PutResumableChunk)
|
||||||
mux.HandleFunc("POST /api/v1/uploads/resumable/{sessionID}/complete", a.CompleteResumableUpload)
|
mux.HandleFunc("POST /api/v1/uploads/resumable/{sessionID}/complete", a.CompleteResumableUpload)
|
||||||
|
mux.HandleFunc("POST /api/v1/uploads/resumable/{sessionID}/complete-uploaded", a.CompleteUploadedResumableUpload)
|
||||||
mux.HandleFunc("DELETE /api/v1/uploads/resumable/{sessionID}", a.CancelResumableUpload)
|
mux.HandleFunc("DELETE /api/v1/uploads/resumable/{sessionID}", a.CancelResumableUpload)
|
||||||
mux.HandleFunc("GET /emoji/{pack}/{file}", a.EmojiAsset)
|
mux.HandleFunc("GET /emoji/{pack}/{file}", a.EmojiAsset)
|
||||||
mux.Handle("GET /static/", a.Static())
|
mux.Handle("GET /static/", a.Static())
|
||||||
|
|||||||
@@ -52,6 +52,7 @@ type fileView struct {
|
|||||||
Reactions []reactionView
|
Reactions []reactionView
|
||||||
ReactionMore int
|
ReactionMore int
|
||||||
Reacted bool
|
Reacted bool
|
||||||
|
Processing bool
|
||||||
}
|
}
|
||||||
|
|
||||||
type reactionView struct {
|
type reactionView struct {
|
||||||
@@ -101,6 +102,15 @@ func (a *App) DownloadPage(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
locked := a.uploadService.IsProtected(box) && !a.isBoxUnlocked(r, box)
|
locked := a.uploadService.IsProtected(box) && !a.isBoxUnlocked(r, box)
|
||||||
|
if isSocialPreviewBot(r) && !locked && len(box.Files) == 1 {
|
||||||
|
if box.Files[0].Processing {
|
||||||
|
http.Error(w, "file is still processing", http.StatusAccepted)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a.serveFileContent(w, r, box, box.Files[0], false)
|
||||||
|
a.logger.Info("single-file box served inline for social preview", withRequestLogAttrs(r, "source", "download", "severity", "user_activity", "code", 2008, "box_id", box.ID, "file_id", box.Files[0].ID)...)
|
||||||
|
return
|
||||||
|
}
|
||||||
visitorID := a.reactionVisitorID(w, r)
|
visitorID := a.reactionVisitorID(w, r)
|
||||||
reactionsByFile, reactedByFile, err := a.reactionService.SummaryForBox(box.ID, visitorID)
|
reactionsByFile, reactedByFile, err := a.reactionService.SummaryForBox(box.ID, visitorID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -159,6 +169,15 @@ func (a *App) DownloadFile(w http.ResponseWriter, r *http.Request) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
locked := a.uploadService.IsProtected(box) && !a.isBoxUnlocked(r, box)
|
locked := a.uploadService.IsProtected(box) && !a.isBoxUnlocked(r, box)
|
||||||
|
if isSocialPreviewBot(r) && !locked {
|
||||||
|
if file.Processing {
|
||||||
|
http.Error(w, "file is still processing", http.StatusAccepted)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a.serveFileContent(w, r, box, file, false)
|
||||||
|
a.logger.Info("file served inline for social preview", withRequestLogAttrs(r, "source", "download", "severity", "user_activity", "code", 2009, "box_id", box.ID, "file_id", file.ID)...)
|
||||||
|
return
|
||||||
|
}
|
||||||
view := a.fileView(box, file)
|
view := a.fileView(box, file)
|
||||||
title := file.Name
|
title := file.Name
|
||||||
description := fmt.Sprintf("%s shared via Warpbox", helpers.FormatBytes(file.Size))
|
description := fmt.Sprintf("%s shared via Warpbox", helpers.FormatBytes(file.Size))
|
||||||
@@ -193,6 +212,10 @@ func (a *App) DownloadFileContent(w http.ResponseWriter, r *http.Request) {
|
|||||||
http.Error(w, "password required", http.StatusUnauthorized)
|
http.Error(w, "password required", http.StatusUnauthorized)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if file.Processing {
|
||||||
|
http.Error(w, "file is still processing", http.StatusAccepted)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
a.serveFileContent(w, r, box, file, r.URL.Query().Get("inline") != "1")
|
a.serveFileContent(w, r, box, file, r.URL.Query().Get("inline") != "1")
|
||||||
a.logger.Info("file content served", withRequestLogAttrs(r, "source", "download", "severity", "user_activity", "code", 2005, "box_id", box.ID, "file_id", file.ID, "attachment", r.URL.Query().Get("inline") != "1")...)
|
a.logger.Info("file content served", withRequestLogAttrs(r, "source", "download", "severity", "user_activity", "code", 2005, "box_id", box.ID, "file_id", file.ID, "attachment", r.URL.Query().Get("inline") != "1")...)
|
||||||
@@ -373,6 +396,7 @@ func (a *App) fileViewWithReactions(box services.Box, file services.File, reacti
|
|||||||
Reactions: reactionViews,
|
Reactions: reactionViews,
|
||||||
ReactionMore: reactionOverflowCount(reactionViews),
|
ReactionMore: reactionOverflowCount(reactionViews),
|
||||||
Reacted: reacted,
|
Reacted: reacted,
|
||||||
|
Processing: file.Processing,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -583,3 +607,31 @@ func absoluteURL(r *http.Request, path string) string {
|
|||||||
}
|
}
|
||||||
return fmt.Sprintf("%s://%s%s", scheme, r.Host, path)
|
return fmt.Sprintf("%s://%s%s", scheme, r.Host, path)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func isSocialPreviewBot(r *http.Request) bool {
|
||||||
|
agent := strings.ToLower(r.UserAgent())
|
||||||
|
if agent == "" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
bots := []string{
|
||||||
|
"discordbot",
|
||||||
|
"twitterbot",
|
||||||
|
"facebookexternalhit",
|
||||||
|
"telegrambot",
|
||||||
|
"whatsapp",
|
||||||
|
"slackbot",
|
||||||
|
"linkedinbot",
|
||||||
|
"skypeuripreview",
|
||||||
|
"embedly",
|
||||||
|
"pinterest",
|
||||||
|
"vkshare",
|
||||||
|
"mattermost",
|
||||||
|
"mastodon",
|
||||||
|
}
|
||||||
|
for _, bot := range bots {
|
||||||
|
if strings.Contains(agent, bot) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package handlers
|
package handlers
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"net/http"
|
"net/http"
|
||||||
@@ -25,6 +26,7 @@ type resumableCreateRequest struct {
|
|||||||
|
|
||||||
type resumableSessionResponse struct {
|
type resumableSessionResponse struct {
|
||||||
SessionID string `json:"sessionId"`
|
SessionID string `json:"sessionId"`
|
||||||
|
ResumeToken string `json:"resumeToken,omitempty"`
|
||||||
ChunkSize int64 `json:"chunkSize"`
|
ChunkSize int64 `json:"chunkSize"`
|
||||||
Status string `json:"status"`
|
Status string `json:"status"`
|
||||||
BoxID string `json:"boxId,omitempty"`
|
BoxID string `json:"boxId,omitempty"`
|
||||||
@@ -33,10 +35,6 @@ type resumableSessionResponse struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (a *App) CreateResumableUpload(w http.ResponseWriter, r *http.Request) {
|
func (a *App) CreateResumableUpload(w http.ResponseWriter, r *http.Request) {
|
||||||
if !a.cfg.ResumableUploadsEnabled {
|
|
||||||
helpers.WriteJSONError(w, http.StatusForbidden, "resumable uploads are disabled")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
user, loggedIn, authErr := a.currentUserWithAuthError(r)
|
user, loggedIn, authErr := a.currentUserWithAuthError(r)
|
||||||
if authErr != nil {
|
if authErr != nil {
|
||||||
a.logger.Warn("resumable upload rejected invalid bearer token", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4011)...)
|
a.logger.Warn("resumable upload rejected invalid bearer token", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4011)...)
|
||||||
@@ -48,6 +46,10 @@ func (a *App) CreateResumableUpload(w http.ResponseWriter, r *http.Request) {
|
|||||||
if !ok {
|
if !ok {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if !settings.ResumableUploadsEnabled {
|
||||||
|
helpers.WriteJSONError(w, http.StatusForbidden, "resumable uploads are disabled")
|
||||||
|
return
|
||||||
|
}
|
||||||
if !loggedIn && !settings.AnonymousUploadsEnabled {
|
if !loggedIn && !settings.AnonymousUploadsEnabled {
|
||||||
a.logger.Warn("resumable anonymous upload rejected disabled", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4013)...)
|
a.logger.Warn("resumable anonymous upload rejected disabled", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4013)...)
|
||||||
helpers.WriteJSONError(w, http.StatusForbidden, "anonymous uploads are disabled")
|
helpers.WriteJSONError(w, http.StatusForbidden, "anonymous uploads are disabled")
|
||||||
@@ -89,7 +91,9 @@ func (a *App) CreateResumableUpload(w http.ResponseWriter, r *http.Request) {
|
|||||||
helpers.WriteJSONError(w, http.StatusRequestEntityTooLarge, err.Error())
|
helpers.WriteJSONError(w, http.StatusRequestEntityTooLarge, err.Error())
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
session, err := a.uploadService.CreateResumableSession(payload.Files, opts, a.cfg.ResumableChunkSize, a.cfg.ResumableRetention)
|
chunkSize := int64(settings.ResumableChunkSizeMB * 1024 * 1024)
|
||||||
|
retention := time.Duration(settings.ResumableRetentionHours) * time.Hour
|
||||||
|
session, err := a.uploadService.CreateResumableSession(payload.Files, opts, chunkSize, retention, resumableChunkRoot(settings))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
a.logger.Warn("resumable session create failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4002, "user_id", user.ID, "error", err.Error())...)
|
a.logger.Warn("resumable session create failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4002, "user_id", user.ID, "error", err.Error())...)
|
||||||
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
|
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
|
||||||
@@ -176,6 +180,17 @@ func (a *App) CompleteResumableUpload(w http.ResponseWriter, r *http.Request) {
|
|||||||
if !ok {
|
if !ok {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if session.Status == services.ResumableStatusCompleted || session.Status == services.ResumableStatusProcessing {
|
||||||
|
result, completed, err := a.uploadService.CompleteResumableSession(r.Context(), session.ID)
|
||||||
|
if err != nil {
|
||||||
|
a.logger.Warn("resumable upload completion replay failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4004, "session_id", session.ID, "error", err.Error())...)
|
||||||
|
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a.logger.Info("resumable upload completion replayed", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2004, "session_id", completed.ID, "box_id", result.BoxID, "files", len(result.Files))...)
|
||||||
|
helpers.WriteJSON(w, http.StatusOK, result)
|
||||||
|
return
|
||||||
|
}
|
||||||
user, loggedIn, _ := a.currentUserWithAuthError(r)
|
user, loggedIn, _ := a.currentUserWithAuthError(r)
|
||||||
isAdminUpload := loggedIn && user.Role == services.UserRoleAdmin
|
isAdminUpload := loggedIn && user.Role == services.UserRoleAdmin
|
||||||
settings, policy, ok := a.loadUploadPolicyForAPI(w, r, user, loggedIn)
|
settings, policy, ok := a.loadUploadPolicyForAPI(w, r, user, loggedIn)
|
||||||
@@ -202,7 +217,7 @@ func (a *App) CompleteResumableUpload(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
result, completed, err := a.uploadService.CompleteResumableSession(r.Context(), session.ID)
|
result, completed, err := a.uploadService.CreateProcessingBoxFromResumable(session.ID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
a.logger.Warn("resumable upload complete failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4004, "session_id", session.ID, "error", err.Error())...)
|
a.logger.Warn("resumable upload complete failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4004, "session_id", session.ID, "error", err.Error())...)
|
||||||
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
|
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
|
||||||
@@ -216,11 +231,90 @@ func (a *App) CompleteResumableUpload(w http.ResponseWriter, r *http.Request) {
|
|||||||
a.logger.Warn("failed to cleanup upload usage", "source", "quota", "severity", "warn", "code", 4405, "error", err.Error())
|
a.logger.Warn("failed to cleanup upload usage", "source", "quota", "severity", "warn", "code", 4405, "error", err.Error())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
jobs.GenerateThumbnailsForBoxAsync(a.uploadService, a.logger, result.BoxID)
|
a.finalizeResumableUploadAsync(completed.ID, result.BoxID)
|
||||||
a.logger.Info("resumable upload completed", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2004, "user_id", user.ID, "session_id", completed.ID, "box_id", result.BoxID, "files", len(result.Files), "bytes", totalBytes, "admin", isAdminUpload, "anonymous", !loggedIn)...)
|
a.logger.Info("resumable upload queued for processing", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2004, "user_id", user.ID, "session_id", completed.ID, "box_id", result.BoxID, "files", len(result.Files), "bytes", totalBytes, "admin", isAdminUpload, "anonymous", !loggedIn)...)
|
||||||
helpers.WriteJSON(w, http.StatusCreated, result)
|
helpers.WriteJSON(w, http.StatusCreated, result)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (a *App) CompleteUploadedResumableUpload(w http.ResponseWriter, r *http.Request) {
|
||||||
|
session, ok := a.authorizedResumableSession(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
user, loggedIn, _ := a.currentUserWithAuthError(r)
|
||||||
|
isAdminUpload := loggedIn && user.Role == services.UserRoleAdmin
|
||||||
|
settings, policy, ok := a.loadUploadPolicyForAPI(w, r, user, loggedIn)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fileSizes := make([]int64, 0, len(session.Files))
|
||||||
|
var totalBytes int64
|
||||||
|
var completeCount int
|
||||||
|
for _, file := range session.Files {
|
||||||
|
if len(file.UploadedChunks) != file.ChunkCount {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fileSizes = append(fileSizes, file.Size)
|
||||||
|
totalBytes += file.Size
|
||||||
|
completeCount++
|
||||||
|
}
|
||||||
|
if completeCount == 0 {
|
||||||
|
helpers.WriteJSONError(w, http.StatusBadRequest, "no fully uploaded files to finish")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !isAdminUpload {
|
||||||
|
if status, message := a.checkUploadPolicyForSizes(r, user, loggedIn, settings, policy, fileSizes, totalBytes); message != "" {
|
||||||
|
helpers.WriteJSONError(w, status, message)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if status, message := a.checkBoxCreationPolicy(r, user, loggedIn, policy); message != "" {
|
||||||
|
helpers.WriteJSONError(w, status, message)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if status, message := a.checkStorageBackendCapacity(session.Options.StorageBackendID, settings, totalBytes); message != "" {
|
||||||
|
helpers.WriteJSONError(w, status, message)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result, completed, err := a.uploadService.CompleteUploadedResumableSession(r.Context(), session.ID)
|
||||||
|
if err != nil {
|
||||||
|
a.logger.Warn("resumable partial complete failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4005, "session_id", session.ID, "error", err.Error())...)
|
||||||
|
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !isAdminUpload {
|
||||||
|
if err := a.recordUploadUsage(r, user, loggedIn, totalBytes, 1); err != nil {
|
||||||
|
a.logger.Warn("failed to record partial resumable upload usage", "source", "quota", "severity", "warn", "code", 4406, "error", err.Error())
|
||||||
|
}
|
||||||
|
if err := a.settingsService.CleanupUsage(time.Now().UTC(), settings.UsageRetentionDays); err != nil {
|
||||||
|
a.logger.Warn("failed to cleanup upload usage", "source", "quota", "severity", "warn", "code", 4405, "error", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
jobs.GenerateThumbnailsForBoxAsync(a.uploadService, a.logger, result.BoxID)
|
||||||
|
a.logger.Info("resumable uploaded files completed", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2007, "user_id", user.ID, "session_id", completed.ID, "box_id", result.BoxID, "files", len(result.Files), "bytes", totalBytes, "admin", isAdminUpload, "anonymous", !loggedIn)...)
|
||||||
|
helpers.WriteJSON(w, http.StatusCreated, result)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *App) finalizeResumableUploadAsync(sessionID, boxID string) {
|
||||||
|
go func() {
|
||||||
|
a.logger.Info("resumable upload processing started", "source", "user-upload", "severity", "user_activity", "code", 2009, "session_id", sessionID, "box_id", boxID)
|
||||||
|
result, err := a.uploadService.FinalizeProcessingResumableSession(context.Background(), sessionID)
|
||||||
|
if err != nil {
|
||||||
|
a.logger.Warn("resumable upload processing failed", "source", "user-upload", "severity", "warn", "code", 4010, "session_id", sessionID, "box_id", boxID, "error", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
jobs.GenerateThumbnailsForBoxAsync(a.uploadService, a.logger, result.BoxID)
|
||||||
|
a.logger.Info("resumable upload processing completed", "source", "user-upload", "severity", "user_activity", "code", 2010, "session_id", sessionID, "box_id", result.BoxID, "files", len(result.Files))
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
func resumableChunkRoot(settings services.UploadPolicySettings) string {
|
||||||
|
if settings.ResumableChunkMode != "custom" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(settings.ResumableChunkPath)
|
||||||
|
}
|
||||||
|
|
||||||
func (a *App) CancelResumableUpload(w http.ResponseWriter, r *http.Request) {
|
func (a *App) CancelResumableUpload(w http.ResponseWriter, r *http.Request) {
|
||||||
session, ok := a.authorizedResumableSession(w, r)
|
session, ok := a.authorizedResumableSession(w, r)
|
||||||
if !ok {
|
if !ok {
|
||||||
@@ -245,6 +339,10 @@ func (a *App) authorizedResumableSession(w http.ResponseWriter, r *http.Request)
|
|||||||
helpers.WriteJSONError(w, http.StatusNotFound, "upload session not found")
|
helpers.WriteJSONError(w, http.StatusNotFound, "upload session not found")
|
||||||
return services.ResumableSession{}, false
|
return services.ResumableSession{}, false
|
||||||
}
|
}
|
||||||
|
if !a.uploadService.VerifyResumableToken(session, r.Header.Get("X-Warpbox-Resume-Token")) {
|
||||||
|
helpers.WriteJSONError(w, http.StatusUnauthorized, "upload session not found")
|
||||||
|
return services.ResumableSession{}, false
|
||||||
|
}
|
||||||
if loggedIn {
|
if loggedIn {
|
||||||
if session.Options.OwnerID != user.ID {
|
if session.Options.OwnerID != user.ID {
|
||||||
helpers.WriteJSONError(w, http.StatusForbidden, "upload session not found")
|
helpers.WriteJSONError(w, http.StatusForbidden, "upload session not found")
|
||||||
@@ -319,6 +417,7 @@ func (a *App) resumableUploadOptions(r *http.Request, payload resumableCreateReq
|
|||||||
func resumableResponse(session services.ResumableSession) resumableSessionResponse {
|
func resumableResponse(session services.ResumableSession) resumableSessionResponse {
|
||||||
return resumableSessionResponse{
|
return resumableSessionResponse{
|
||||||
SessionID: session.ID,
|
SessionID: session.ID,
|
||||||
|
ResumeToken: session.ResumeToken,
|
||||||
ChunkSize: session.ChunkSize,
|
ChunkSize: session.ChunkSize,
|
||||||
Status: session.Status,
|
Status: session.Status,
|
||||||
BoxID: session.BoxID,
|
BoxID: session.BoxID,
|
||||||
|
|||||||
@@ -106,6 +106,51 @@ func TestUploadTextResponseReturnsOnlyBoxURL(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestSocialPreviewBotGetsRawSingleFileBox(t *testing.T) {
|
||||||
|
app, cleanup := newTestApp(t)
|
||||||
|
defer cleanup()
|
||||||
|
payload := uploadThroughApp(t, app)
|
||||||
|
|
||||||
|
request := httptest.NewRequest(http.MethodGet, "/d/"+payload.BoxID, nil)
|
||||||
|
request.Header.Set("User-Agent", "Discordbot/2.0")
|
||||||
|
request.SetPathValue("boxID", payload.BoxID)
|
||||||
|
response := httptest.NewRecorder()
|
||||||
|
app.DownloadPage(response, request)
|
||||||
|
|
||||||
|
if response.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, body = %s", response.Code, response.Body.String())
|
||||||
|
}
|
||||||
|
if strings.Contains(response.Body.String(), "Shared files on Warpbox") {
|
||||||
|
t.Fatalf("social preview bot received HTML download page")
|
||||||
|
}
|
||||||
|
if response.Body.String() != "hello" {
|
||||||
|
t.Fatalf("social preview body = %q", response.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSocialPreviewBotGetsRawFilePreview(t *testing.T) {
|
||||||
|
app, cleanup := newTestApp(t)
|
||||||
|
defer cleanup()
|
||||||
|
payload := uploadThroughApp(t, app)
|
||||||
|
|
||||||
|
request := httptest.NewRequest(http.MethodGet, "/d/"+payload.BoxID+"/f/"+payload.Files[0].ID, nil)
|
||||||
|
request.Header.Set("User-Agent", "TelegramBot")
|
||||||
|
request.SetPathValue("boxID", payload.BoxID)
|
||||||
|
request.SetPathValue("fileID", payload.Files[0].ID)
|
||||||
|
response := httptest.NewRecorder()
|
||||||
|
app.DownloadFile(response, request)
|
||||||
|
|
||||||
|
if response.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, body = %s", response.Code, response.Body.String())
|
||||||
|
}
|
||||||
|
if strings.Contains(response.Body.String(), "preview-title") {
|
||||||
|
t.Fatalf("social preview bot received HTML preview page")
|
||||||
|
}
|
||||||
|
if response.Body.String() != "hello" {
|
||||||
|
t.Fatalf("social preview body = %q", response.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResumableUploadFlowCreatesNormalBox(t *testing.T) {
|
func TestResumableUploadFlowCreatesNormalBox(t *testing.T) {
|
||||||
app, cleanup := newTestApp(t)
|
app, cleanup := newTestApp(t)
|
||||||
defer cleanup()
|
defer cleanup()
|
||||||
@@ -120,6 +165,7 @@ func TestResumableUploadFlowCreatesNormalBox(t *testing.T) {
|
|||||||
}
|
}
|
||||||
var session struct {
|
var session struct {
|
||||||
SessionID string `json:"sessionId"`
|
SessionID string `json:"sessionId"`
|
||||||
|
ResumeToken string `json:"resumeToken"`
|
||||||
ChunkSize int64 `json:"chunkSize"`
|
ChunkSize int64 `json:"chunkSize"`
|
||||||
Files []struct {
|
Files []struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
@@ -130,7 +176,7 @@ func TestResumableUploadFlowCreatesNormalBox(t *testing.T) {
|
|||||||
if err := json.Unmarshal(createResponse.Body.Bytes(), &session); err != nil {
|
if err := json.Unmarshal(createResponse.Body.Bytes(), &session); err != nil {
|
||||||
t.Fatalf("json.Unmarshal session returned error: %v", err)
|
t.Fatalf("json.Unmarshal session returned error: %v", err)
|
||||||
}
|
}
|
||||||
if session.SessionID == "" || session.ChunkSize != 4 || len(session.Files) != 1 || session.Files[0].ChunkCount != 3 {
|
if session.SessionID == "" || session.ResumeToken == "" || session.ChunkSize != 4 || len(session.Files) != 1 || session.Files[0].ChunkCount != 3 {
|
||||||
t.Fatalf("unexpected session response: %+v", session)
|
t.Fatalf("unexpected session response: %+v", session)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -140,6 +186,7 @@ func TestResumableUploadFlowCreatesNormalBox(t *testing.T) {
|
|||||||
request.SetPathValue("sessionID", session.SessionID)
|
request.SetPathValue("sessionID", session.SessionID)
|
||||||
request.SetPathValue("fileID", session.Files[0].ID)
|
request.SetPathValue("fileID", session.Files[0].ID)
|
||||||
request.SetPathValue("index", strconv.Itoa(index))
|
request.SetPathValue("index", strconv.Itoa(index))
|
||||||
|
request.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
|
||||||
response := httptest.NewRecorder()
|
response := httptest.NewRecorder()
|
||||||
app.PutResumableChunk(response, request)
|
app.PutResumableChunk(response, request)
|
||||||
if response.Code != http.StatusOK {
|
if response.Code != http.StatusOK {
|
||||||
@@ -149,6 +196,7 @@ func TestResumableUploadFlowCreatesNormalBox(t *testing.T) {
|
|||||||
|
|
||||||
completeRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
|
completeRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
|
||||||
completeRequest.SetPathValue("sessionID", session.SessionID)
|
completeRequest.SetPathValue("sessionID", session.SessionID)
|
||||||
|
completeRequest.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
|
||||||
completeResponse := httptest.NewRecorder()
|
completeResponse := httptest.NewRecorder()
|
||||||
app.CompleteResumableUpload(completeResponse, completeRequest)
|
app.CompleteResumableUpload(completeResponse, completeRequest)
|
||||||
if completeResponse.Code != http.StatusCreated {
|
if completeResponse.Code != http.StatusCreated {
|
||||||
@@ -158,10 +206,22 @@ func TestResumableUploadFlowCreatesNormalBox(t *testing.T) {
|
|||||||
if err := json.Unmarshal(completeResponse.Body.Bytes(), &payload); err != nil {
|
if err := json.Unmarshal(completeResponse.Body.Bytes(), &payload); err != nil {
|
||||||
t.Fatalf("json.Unmarshal result returned error: %v", err)
|
t.Fatalf("json.Unmarshal result returned error: %v", err)
|
||||||
}
|
}
|
||||||
box, err := app.uploadService.GetBox(payload.BoxID)
|
replayRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
|
||||||
if err != nil {
|
replayRequest.SetPathValue("sessionID", session.SessionID)
|
||||||
t.Fatalf("GetBox returned error: %v", err)
|
replayRequest.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
|
||||||
|
replayResponse := httptest.NewRecorder()
|
||||||
|
app.CompleteResumableUpload(replayResponse, replayRequest)
|
||||||
|
if replayResponse.Code != http.StatusOK {
|
||||||
|
t.Fatalf("complete replay status = %d, body = %s", replayResponse.Code, replayResponse.Body.String())
|
||||||
}
|
}
|
||||||
|
var replayPayload services.UploadResult
|
||||||
|
if err := json.Unmarshal(replayResponse.Body.Bytes(), &replayPayload); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal replay result returned error: %v", err)
|
||||||
|
}
|
||||||
|
if replayPayload.BoxID != payload.BoxID || replayPayload.BoxURL == "" {
|
||||||
|
t.Fatalf("unexpected replay result: %+v, original: %+v", replayPayload, payload)
|
||||||
|
}
|
||||||
|
box := waitForProcessedBox(t, app, payload.BoxID)
|
||||||
if len(box.Files) != 1 || box.Files[0].Name != "note.txt" || box.Files[0].Size != 11 {
|
if len(box.Files) != 1 || box.Files[0].Name != "note.txt" || box.Files[0].Size != 11 {
|
||||||
t.Fatalf("unexpected box files: %+v", box.Files)
|
t.Fatalf("unexpected box files: %+v", box.Files)
|
||||||
}
|
}
|
||||||
@@ -191,6 +251,7 @@ func TestResumableUploadRequiresAllChunks(t *testing.T) {
|
|||||||
}
|
}
|
||||||
var session struct {
|
var session struct {
|
||||||
SessionID string `json:"sessionId"`
|
SessionID string `json:"sessionId"`
|
||||||
|
ResumeToken string `json:"resumeToken"`
|
||||||
Files []struct {
|
Files []struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
} `json:"files"`
|
} `json:"files"`
|
||||||
@@ -202,6 +263,7 @@ func TestResumableUploadRequiresAllChunks(t *testing.T) {
|
|||||||
chunkRequest.SetPathValue("sessionID", session.SessionID)
|
chunkRequest.SetPathValue("sessionID", session.SessionID)
|
||||||
chunkRequest.SetPathValue("fileID", session.Files[0].ID)
|
chunkRequest.SetPathValue("fileID", session.Files[0].ID)
|
||||||
chunkRequest.SetPathValue("index", "0")
|
chunkRequest.SetPathValue("index", "0")
|
||||||
|
chunkRequest.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
|
||||||
chunkResponse := httptest.NewRecorder()
|
chunkResponse := httptest.NewRecorder()
|
||||||
app.PutResumableChunk(chunkResponse, chunkRequest)
|
app.PutResumableChunk(chunkResponse, chunkRequest)
|
||||||
if chunkResponse.Code != http.StatusOK {
|
if chunkResponse.Code != http.StatusOK {
|
||||||
@@ -210,6 +272,7 @@ func TestResumableUploadRequiresAllChunks(t *testing.T) {
|
|||||||
|
|
||||||
completeRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
|
completeRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
|
||||||
completeRequest.SetPathValue("sessionID", session.SessionID)
|
completeRequest.SetPathValue("sessionID", session.SessionID)
|
||||||
|
completeRequest.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
|
||||||
completeResponse := httptest.NewRecorder()
|
completeResponse := httptest.NewRecorder()
|
||||||
app.CompleteResumableUpload(completeResponse, completeRequest)
|
app.CompleteResumableUpload(completeResponse, completeRequest)
|
||||||
if completeResponse.Code != http.StatusBadRequest {
|
if completeResponse.Code != http.StatusBadRequest {
|
||||||
@@ -217,6 +280,54 @@ func TestResumableUploadRequiresAllChunks(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestResumableStatusRequiresResumeToken(t *testing.T) {
|
||||||
|
app, cleanup := newTestApp(t)
|
||||||
|
defer cleanup()
|
||||||
|
|
||||||
|
createRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable", strings.NewReader(`{"files":[{"name":"note.txt","size":4,"contentType":"text/plain"}]}`))
|
||||||
|
createResponse := httptest.NewRecorder()
|
||||||
|
app.CreateResumableUpload(createResponse, createRequest)
|
||||||
|
if createResponse.Code != http.StatusCreated {
|
||||||
|
t.Fatalf("create status = %d, body = %s", createResponse.Code, createResponse.Body.String())
|
||||||
|
}
|
||||||
|
var session struct {
|
||||||
|
SessionID string `json:"sessionId"`
|
||||||
|
ResumeToken string `json:"resumeToken"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(createResponse.Body.Bytes(), &session); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal session returned error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
missing := httptest.NewRequest(http.MethodGet, "/api/v1/uploads/resumable/"+session.SessionID, nil)
|
||||||
|
missing.SetPathValue("sessionID", session.SessionID)
|
||||||
|
missingResponse := httptest.NewRecorder()
|
||||||
|
app.ResumableUploadStatus(missingResponse, missing)
|
||||||
|
if missingResponse.Code != http.StatusUnauthorized {
|
||||||
|
t.Fatalf("missing token status = %d, body = %s", missingResponse.Code, missingResponse.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
wrong := httptest.NewRequest(http.MethodGet, "/api/v1/uploads/resumable/"+session.SessionID, nil)
|
||||||
|
wrong.SetPathValue("sessionID", session.SessionID)
|
||||||
|
wrong.Header.Set("X-Warpbox-Resume-Token", "wrong")
|
||||||
|
wrongResponse := httptest.NewRecorder()
|
||||||
|
app.ResumableUploadStatus(wrongResponse, wrong)
|
||||||
|
if wrongResponse.Code != http.StatusUnauthorized {
|
||||||
|
t.Fatalf("wrong token status = %d, body = %s", wrongResponse.Code, wrongResponse.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
valid := httptest.NewRequest(http.MethodGet, "/api/v1/uploads/resumable/"+session.SessionID, nil)
|
||||||
|
valid.SetPathValue("sessionID", session.SessionID)
|
||||||
|
valid.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
|
||||||
|
validResponse := httptest.NewRecorder()
|
||||||
|
app.ResumableUploadStatus(validResponse, valid)
|
||||||
|
if validResponse.Code != http.StatusOK {
|
||||||
|
t.Fatalf("valid token status = %d, body = %s", validResponse.Code, validResponse.Body.String())
|
||||||
|
}
|
||||||
|
if strings.Contains(validResponse.Body.String(), "resumeTokenHash") {
|
||||||
|
t.Fatalf("status response leaked token hash: %s", validResponse.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResumableUploadCanAddFilesToExistingSession(t *testing.T) {
|
func TestResumableUploadCanAddFilesToExistingSession(t *testing.T) {
|
||||||
app, cleanup := newTestApp(t)
|
app, cleanup := newTestApp(t)
|
||||||
defer cleanup()
|
defer cleanup()
|
||||||
@@ -230,6 +341,7 @@ func TestResumableUploadCanAddFilesToExistingSession(t *testing.T) {
|
|||||||
}
|
}
|
||||||
var session struct {
|
var session struct {
|
||||||
SessionID string `json:"sessionId"`
|
SessionID string `json:"sessionId"`
|
||||||
|
ResumeToken string `json:"resumeToken"`
|
||||||
Files []struct {
|
Files []struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
} `json:"files"`
|
} `json:"files"`
|
||||||
@@ -241,6 +353,7 @@ func TestResumableUploadCanAddFilesToExistingSession(t *testing.T) {
|
|||||||
firstChunk.SetPathValue("sessionID", session.SessionID)
|
firstChunk.SetPathValue("sessionID", session.SessionID)
|
||||||
firstChunk.SetPathValue("fileID", session.Files[0].ID)
|
firstChunk.SetPathValue("fileID", session.Files[0].ID)
|
||||||
firstChunk.SetPathValue("index", "0")
|
firstChunk.SetPathValue("index", "0")
|
||||||
|
firstChunk.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
|
||||||
firstChunkResponse := httptest.NewRecorder()
|
firstChunkResponse := httptest.NewRecorder()
|
||||||
app.PutResumableChunk(firstChunkResponse, firstChunk)
|
app.PutResumableChunk(firstChunkResponse, firstChunk)
|
||||||
if firstChunkResponse.Code != http.StatusOK {
|
if firstChunkResponse.Code != http.StatusOK {
|
||||||
@@ -249,6 +362,7 @@ func TestResumableUploadCanAddFilesToExistingSession(t *testing.T) {
|
|||||||
|
|
||||||
addRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/files", strings.NewReader(`{"files":[{"name":"two.txt","size":4,"contentType":"text/plain","fingerprint":"two"}]}`))
|
addRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/files", strings.NewReader(`{"files":[{"name":"two.txt","size":4,"contentType":"text/plain","fingerprint":"two"}]}`))
|
||||||
addRequest.SetPathValue("sessionID", session.SessionID)
|
addRequest.SetPathValue("sessionID", session.SessionID)
|
||||||
|
addRequest.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
|
||||||
addResponse := httptest.NewRecorder()
|
addResponse := httptest.NewRecorder()
|
||||||
app.AddResumableFiles(addResponse, addRequest)
|
app.AddResumableFiles(addResponse, addRequest)
|
||||||
if addResponse.Code != http.StatusOK {
|
if addResponse.Code != http.StatusOK {
|
||||||
@@ -271,6 +385,7 @@ func TestResumableUploadCanAddFilesToExistingSession(t *testing.T) {
|
|||||||
secondChunk.SetPathValue("sessionID", session.SessionID)
|
secondChunk.SetPathValue("sessionID", session.SessionID)
|
||||||
secondChunk.SetPathValue("fileID", updated.Files[1].ID)
|
secondChunk.SetPathValue("fileID", updated.Files[1].ID)
|
||||||
secondChunk.SetPathValue("index", "0")
|
secondChunk.SetPathValue("index", "0")
|
||||||
|
secondChunk.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
|
||||||
secondChunkResponse := httptest.NewRecorder()
|
secondChunkResponse := httptest.NewRecorder()
|
||||||
app.PutResumableChunk(secondChunkResponse, secondChunk)
|
app.PutResumableChunk(secondChunkResponse, secondChunk)
|
||||||
if secondChunkResponse.Code != http.StatusOK {
|
if secondChunkResponse.Code != http.StatusOK {
|
||||||
@@ -278,6 +393,7 @@ func TestResumableUploadCanAddFilesToExistingSession(t *testing.T) {
|
|||||||
}
|
}
|
||||||
completeRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
|
completeRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
|
||||||
completeRequest.SetPathValue("sessionID", session.SessionID)
|
completeRequest.SetPathValue("sessionID", session.SessionID)
|
||||||
|
completeRequest.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
|
||||||
completeResponse := httptest.NewRecorder()
|
completeResponse := httptest.NewRecorder()
|
||||||
app.CompleteResumableUpload(completeResponse, completeRequest)
|
app.CompleteResumableUpload(completeResponse, completeRequest)
|
||||||
if completeResponse.Code != http.StatusCreated {
|
if completeResponse.Code != http.StatusCreated {
|
||||||
@@ -296,6 +412,79 @@ func TestResumableUploadCanAddFilesToExistingSession(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestResumableCompleteUploadedRequiresTokenAndKeepsFinishedFiles(t *testing.T) {
|
||||||
|
app, cleanup := newTestApp(t)
|
||||||
|
defer cleanup()
|
||||||
|
|
||||||
|
createBody := `{"files":[{"name":"done.txt","size":4,"contentType":"text/plain","fingerprint":"done"},{"name":"partial.txt","size":8,"contentType":"text/plain","fingerprint":"partial"}],"expiresMinutes":60}`
|
||||||
|
createRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable", strings.NewReader(createBody))
|
||||||
|
createResponse := httptest.NewRecorder()
|
||||||
|
app.CreateResumableUpload(createResponse, createRequest)
|
||||||
|
if createResponse.Code != http.StatusCreated {
|
||||||
|
t.Fatalf("create status = %d, body = %s", createResponse.Code, createResponse.Body.String())
|
||||||
|
}
|
||||||
|
var session struct {
|
||||||
|
SessionID string `json:"sessionId"`
|
||||||
|
ResumeToken string `json:"resumeToken"`
|
||||||
|
Files []struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
} `json:"files"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(createResponse.Body.Bytes(), &session); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal session returned error: %v", err)
|
||||||
|
}
|
||||||
|
for _, chunk := range []struct {
|
||||||
|
fileIndex int
|
||||||
|
index string
|
||||||
|
body string
|
||||||
|
}{
|
||||||
|
{fileIndex: 0, index: "0", body: "done"},
|
||||||
|
{fileIndex: 1, index: "0", body: "part"},
|
||||||
|
} {
|
||||||
|
request := httptest.NewRequest(http.MethodPut, "/api/v1/uploads/resumable/"+session.SessionID+"/files/"+session.Files[chunk.fileIndex].ID+"/chunks/"+chunk.index, strings.NewReader(chunk.body))
|
||||||
|
request.SetPathValue("sessionID", session.SessionID)
|
||||||
|
request.SetPathValue("fileID", session.Files[chunk.fileIndex].ID)
|
||||||
|
request.SetPathValue("index", chunk.index)
|
||||||
|
request.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
|
||||||
|
response := httptest.NewRecorder()
|
||||||
|
app.PutResumableChunk(response, request)
|
||||||
|
if response.Code != http.StatusOK {
|
||||||
|
t.Fatalf("chunk status = %d, body = %s", response.Code, response.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
missing := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete-uploaded", nil)
|
||||||
|
missing.SetPathValue("sessionID", session.SessionID)
|
||||||
|
missingResponse := httptest.NewRecorder()
|
||||||
|
app.CompleteUploadedResumableUpload(missingResponse, missing)
|
||||||
|
if missingResponse.Code != http.StatusUnauthorized {
|
||||||
|
t.Fatalf("missing token status = %d, body = %s", missingResponse.Code, missingResponse.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
complete := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete-uploaded", nil)
|
||||||
|
complete.SetPathValue("sessionID", session.SessionID)
|
||||||
|
complete.Header.Set("X-Warpbox-Resume-Token", session.ResumeToken)
|
||||||
|
completeResponse := httptest.NewRecorder()
|
||||||
|
app.CompleteUploadedResumableUpload(completeResponse, complete)
|
||||||
|
if completeResponse.Code != http.StatusCreated {
|
||||||
|
t.Fatalf("complete-uploaded status = %d, body = %s", completeResponse.Code, completeResponse.Body.String())
|
||||||
|
}
|
||||||
|
var payload services.UploadResult
|
||||||
|
if err := json.Unmarshal(completeResponse.Body.Bytes(), &payload); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal result returned error: %v", err)
|
||||||
|
}
|
||||||
|
box, err := app.uploadService.GetBox(payload.BoxID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetBox returned error: %v", err)
|
||||||
|
}
|
||||||
|
if len(box.Files) != 1 || box.Files[0].Name != "done.txt" {
|
||||||
|
t.Fatalf("complete-uploaded box files = %+v", box.Files)
|
||||||
|
}
|
||||||
|
if _, err := app.uploadService.GetResumableSession(session.SessionID); !os.IsNotExist(err) {
|
||||||
|
t.Fatalf("GetResumableSession after complete-uploaded error = %v, want os.ErrNotExist", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestManageBoxAndDeleteFlow(t *testing.T) {
|
func TestManageBoxAndDeleteFlow(t *testing.T) {
|
||||||
app, cleanup := newTestApp(t)
|
app, cleanup := newTestApp(t)
|
||||||
defer cleanup()
|
defer cleanup()
|
||||||
@@ -424,6 +613,10 @@ func newTestApp(t *testing.T) (*App, func()) {
|
|||||||
UserDailyUploadMB: 8,
|
UserDailyUploadMB: 8,
|
||||||
DefaultUserStorageMB: 16,
|
DefaultUserStorageMB: 16,
|
||||||
UsageRetentionDays: 30,
|
UsageRetentionDays: 30,
|
||||||
|
ResumableUploadsEnabled: true,
|
||||||
|
ResumableChunkSizeMB: 0.000003814697265625,
|
||||||
|
ResumableRetentionHours: 1,
|
||||||
|
ResumableChunkMode: "same",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
service, err := services.NewUploadService(cfg.MaxUploadSize, cfg.DataDir, cfg.BaseURL, logger)
|
service, err := services.NewUploadService(cfg.MaxUploadSize, cfg.DataDir, cfg.BaseURL, logger)
|
||||||
@@ -538,6 +731,31 @@ func tokenFromURL(t *testing.T, value string) string {
|
|||||||
return parts[len(parts)-1]
|
return parts[len(parts)-1]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func waitForProcessedBox(t *testing.T, app *App, boxID string) services.Box {
|
||||||
|
t.Helper()
|
||||||
|
var box services.Box
|
||||||
|
for i := 0; i < 50; i++ {
|
||||||
|
next, err := app.uploadService.GetBox(boxID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetBox returned error: %v", err)
|
||||||
|
}
|
||||||
|
box = next
|
||||||
|
processing := false
|
||||||
|
for _, file := range box.Files {
|
||||||
|
if file.Processing {
|
||||||
|
processing = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !processing {
|
||||||
|
return box
|
||||||
|
}
|
||||||
|
time.Sleep(10 * time.Millisecond)
|
||||||
|
}
|
||||||
|
t.Fatalf("box %s was still processing: %+v", boxID, box.Files)
|
||||||
|
return box
|
||||||
|
}
|
||||||
|
|
||||||
func copyDir(src, dst string) error {
|
func copyDir(src, dst string) error {
|
||||||
return filepath.WalkDir(src, func(path string, d os.DirEntry, err error) error {
|
return filepath.WalkDir(src, func(path string, d os.DirEntry, err error) error {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -2,6 +2,9 @@ package services
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"crypto/subtle"
|
||||||
|
"encoding/hex"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
@@ -18,6 +21,7 @@ var resumableUploadsBucket = []byte("resumable_uploads")
|
|||||||
|
|
||||||
const (
|
const (
|
||||||
ResumableStatusUploading = "uploading"
|
ResumableStatusUploading = "uploading"
|
||||||
|
ResumableStatusProcessing = "processing"
|
||||||
ResumableStatusCompleted = "completed"
|
ResumableStatusCompleted = "completed"
|
||||||
ResumableStatusCancelled = "cancelled"
|
ResumableStatusCancelled = "cancelled"
|
||||||
)
|
)
|
||||||
@@ -36,6 +40,9 @@ type ResumableSession struct {
|
|||||||
ChunkSize int64 `json:"chunkSize"`
|
ChunkSize int64 `json:"chunkSize"`
|
||||||
Status string `json:"status"`
|
Status string `json:"status"`
|
||||||
BoxID string `json:"boxId,omitempty"`
|
BoxID string `json:"boxId,omitempty"`
|
||||||
|
ResumeTokenHash string `json:"resumeTokenHash,omitempty"`
|
||||||
|
ResumeToken string `json:"-"`
|
||||||
|
ChunkRoot string `json:"chunkRoot,omitempty"`
|
||||||
CreatedAt time.Time `json:"createdAt"`
|
CreatedAt time.Time `json:"createdAt"`
|
||||||
UpdatedAt time.Time `json:"updatedAt"`
|
UpdatedAt time.Time `json:"updatedAt"`
|
||||||
ExpiresAt time.Time `json:"expiresAt"`
|
ExpiresAt time.Time `json:"expiresAt"`
|
||||||
@@ -58,7 +65,7 @@ func (s *UploadService) ensureResumableBucket() error {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *UploadService) CreateResumableSession(files []ResumableFileInput, opts UploadOptions, chunkSize int64, retention time.Duration) (ResumableSession, error) {
|
func (s *UploadService) CreateResumableSession(files []ResumableFileInput, opts UploadOptions, chunkSize int64, retention time.Duration, chunkRoot string) (ResumableSession, error) {
|
||||||
if len(files) == 0 {
|
if len(files) == 0 {
|
||||||
return ResumableSession{}, fmt.Errorf("no files were uploaded")
|
return ResumableSession{}, fmt.Errorf("no files were uploaded")
|
||||||
}
|
}
|
||||||
@@ -77,12 +84,17 @@ func (s *UploadService) CreateResumableSession(files []ResumableFileInput, opts
|
|||||||
return ResumableSession{}, err
|
return ResumableSession{}, err
|
||||||
}
|
}
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
|
resumeToken := randomID(32)
|
||||||
|
sessionID := randomID(12)
|
||||||
session := ResumableSession{
|
session := ResumableSession{
|
||||||
ID: randomID(12),
|
ID: sessionID,
|
||||||
Options: opts,
|
Options: opts,
|
||||||
Files: sessionFiles,
|
Files: sessionFiles,
|
||||||
ChunkSize: chunkSize,
|
ChunkSize: chunkSize,
|
||||||
Status: ResumableStatusUploading,
|
Status: ResumableStatusUploading,
|
||||||
|
ResumeTokenHash: resumableTokenHash(sessionID, resumeToken),
|
||||||
|
ResumeToken: resumeToken,
|
||||||
|
ChunkRoot: strings.TrimSpace(chunkRoot),
|
||||||
CreatedAt: now,
|
CreatedAt: now,
|
||||||
UpdatedAt: now,
|
UpdatedAt: now,
|
||||||
ExpiresAt: now.Add(retention),
|
ExpiresAt: now.Add(retention),
|
||||||
@@ -93,6 +105,14 @@ func (s *UploadService) CreateResumableSession(files []ResumableFileInput, opts
|
|||||||
return session, nil
|
return session, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) VerifyResumableToken(session ResumableSession, token string) bool {
|
||||||
|
if session.ResumeTokenHash == "" || strings.TrimSpace(token) == "" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
hash := resumableTokenHash(session.ID, token)
|
||||||
|
return subtle.ConstantTimeCompare([]byte(hash), []byte(session.ResumeTokenHash)) == 1
|
||||||
|
}
|
||||||
|
|
||||||
func (s *UploadService) AddResumableFiles(sessionID string, files []ResumableFileInput) (ResumableSession, error) {
|
func (s *UploadService) AddResumableFiles(sessionID string, files []ResumableFileInput) (ResumableSession, error) {
|
||||||
if len(files) == 0 {
|
if len(files) == 0 {
|
||||||
return s.GetResumableSession(sessionID)
|
return s.GetResumableSession(sessionID)
|
||||||
@@ -165,11 +185,11 @@ func (s *UploadService) PutResumableChunk(ctx context.Context, sessionID, fileID
|
|||||||
return ResumableSession{}, fmt.Errorf("chunk index is invalid")
|
return ResumableSession{}, fmt.Errorf("chunk index is invalid")
|
||||||
}
|
}
|
||||||
expectedSize := expectedChunkSize(file.Size, session.ChunkSize, index)
|
expectedSize := expectedChunkSize(file.Size, session.ChunkSize, index)
|
||||||
chunkDir := s.resumableFileDir(session.ID, file.ID)
|
chunkDir := s.resumableFileDirFor(session, file.ID)
|
||||||
if err := os.MkdirAll(chunkDir, 0o755); err != nil {
|
if err := os.MkdirAll(chunkDir, 0o755); err != nil {
|
||||||
return ResumableSession{}, err
|
return ResumableSession{}, err
|
||||||
}
|
}
|
||||||
chunkPath := s.resumableChunkPath(session.ID, file.ID, index)
|
chunkPath := s.resumableChunkPathFor(session, file.ID, index)
|
||||||
tempPath := chunkPath + ".tmp"
|
tempPath := chunkPath + ".tmp"
|
||||||
target, err := os.OpenFile(tempPath, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
|
target, err := os.OpenFile(tempPath, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -206,20 +226,26 @@ func (s *UploadService) CompleteResumableSession(ctx context.Context, sessionID
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return UploadResult{}, ResumableSession{}, err
|
return UploadResult{}, ResumableSession{}, err
|
||||||
}
|
}
|
||||||
|
if (session.Status == ResumableStatusCompleted || session.Status == ResumableStatusProcessing) && session.BoxID != "" {
|
||||||
|
box, err := s.GetBox(session.BoxID)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
return s.resultForBox(box, ""), session, nil
|
||||||
|
}
|
||||||
if err := resumableSessionWritable(session); err != nil {
|
if err := resumableSessionWritable(session); err != nil {
|
||||||
return UploadResult{}, ResumableSession{}, err
|
return UploadResult{}, ResumableSession{}, err
|
||||||
}
|
}
|
||||||
staged, cleanup, err := s.assembleResumableFiles(ctx, session)
|
staged, err := s.resumableIncomingFiles(session)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return UploadResult{}, ResumableSession{}, err
|
return UploadResult{}, ResumableSession{}, err
|
||||||
}
|
}
|
||||||
defer cleanup()
|
|
||||||
|
|
||||||
result, err := s.CreateBoxFromIncoming(staged, session.Options)
|
result, err := s.CreateBoxFromIncomingContext(ctx, staged, session.Options)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return UploadResult{}, ResumableSession{}, err
|
return UploadResult{}, ResumableSession{}, err
|
||||||
}
|
}
|
||||||
if err := os.RemoveAll(s.resumableSessionDir(session.ID)); err != nil {
|
if err := os.RemoveAll(s.resumableSessionDirFor(session)); err != nil {
|
||||||
return UploadResult{}, ResumableSession{}, err
|
return UploadResult{}, ResumableSession{}, err
|
||||||
}
|
}
|
||||||
session.Status = ResumableStatusCompleted
|
session.Status = ResumableStatusCompleted
|
||||||
@@ -231,17 +257,205 @@ func (s *UploadService) CompleteResumableSession(ctx context.Context, sessionID
|
|||||||
return result, session, nil
|
return result, session, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) CreateProcessingBoxFromResumable(sessionID string) (UploadResult, ResumableSession, error) {
|
||||||
|
session, err := s.GetResumableSession(sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
if (session.Status == ResumableStatusCompleted || session.Status == ResumableStatusProcessing) && session.BoxID != "" {
|
||||||
|
box, err := s.GetBox(session.BoxID)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
return s.resultForBox(box, ""), session, nil
|
||||||
|
}
|
||||||
|
if err := resumableSessionWritable(session); err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
if _, err := s.resumableIncomingFiles(session); err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
now := time.Now().UTC()
|
||||||
|
expiresAt := now.AddDate(0, 0, 7)
|
||||||
|
if session.Options.ExpiresInMinutes < 0 || session.Options.MaxDays < 0 {
|
||||||
|
expiresAt = now.AddDate(100, 0, 0)
|
||||||
|
} else if session.Options.ExpiresInMinutes > 0 {
|
||||||
|
expiresAt = now.Add(time.Duration(session.Options.ExpiresInMinutes) * time.Minute)
|
||||||
|
} else if session.Options.MaxDays > 0 {
|
||||||
|
expiresAt = now.Add(time.Duration(session.Options.MaxDays) * 24 * time.Hour)
|
||||||
|
}
|
||||||
|
|
||||||
|
box := Box{
|
||||||
|
ID: randomID(10),
|
||||||
|
OwnerID: strings.TrimSpace(session.Options.OwnerID),
|
||||||
|
CollectionID: strings.TrimSpace(session.Options.CollectionID),
|
||||||
|
CreatorIP: strings.TrimSpace(session.Options.CreatorIP),
|
||||||
|
StorageBackendID: normalizeBackendID(session.Options.StorageBackendID),
|
||||||
|
CreatedAt: now,
|
||||||
|
ExpiresAt: expiresAt,
|
||||||
|
MaxDownloads: session.Options.MaxDownloads,
|
||||||
|
Obfuscate: session.Options.ObfuscateMetadata && (strings.TrimSpace(session.Options.Password) != "" || strings.TrimSpace(session.Options.PasswordHash) != ""),
|
||||||
|
Files: make([]File, 0, len(session.Files)),
|
||||||
|
}
|
||||||
|
deleteToken := randomID(32)
|
||||||
|
box.DeleteTokenHash = deleteTokenHash(box.ID, deleteToken)
|
||||||
|
if strings.TrimSpace(session.Options.PasswordHash) != "" {
|
||||||
|
box.PasswordSalt = session.Options.PasswordSalt
|
||||||
|
box.PasswordHash = session.Options.PasswordHash
|
||||||
|
} else if strings.TrimSpace(session.Options.Password) != "" {
|
||||||
|
salt, hash := hashPassword(session.Options.Password)
|
||||||
|
box.PasswordSalt = salt
|
||||||
|
box.PasswordHash = hash
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, incoming := range session.Files {
|
||||||
|
fileID := randomID(8)
|
||||||
|
storedName := "@each@" + fileID + strings.ToLower(filepath.Ext(incoming.Name))
|
||||||
|
objectKey := boxObjectKey(box.ID, storedName)
|
||||||
|
contentType := incoming.ContentType
|
||||||
|
if contentType == "" {
|
||||||
|
contentType = "application/octet-stream"
|
||||||
|
}
|
||||||
|
box.Files = append(box.Files, File{
|
||||||
|
ID: fileID,
|
||||||
|
Name: filepath.Base(incoming.Name),
|
||||||
|
StoredName: storedName,
|
||||||
|
Size: incoming.Size,
|
||||||
|
ContentType: contentType,
|
||||||
|
PreviewKind: previewKind(contentType),
|
||||||
|
ObjectKey: objectKey,
|
||||||
|
Processing: true,
|
||||||
|
UploadedAt: now,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if err := s.saveBoxRecord(box); err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
session.Status = ResumableStatusProcessing
|
||||||
|
session.BoxID = box.ID
|
||||||
|
session.UpdatedAt = time.Now().UTC()
|
||||||
|
if err := s.saveResumableSession(session); err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
return s.resultForBox(box, deleteToken), session, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) FinalizeProcessingResumableSession(ctx context.Context, sessionID string) (UploadResult, error) {
|
||||||
|
session, err := s.GetResumableSession(sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, err
|
||||||
|
}
|
||||||
|
if session.Status == ResumableStatusCompleted && session.BoxID != "" {
|
||||||
|
box, err := s.GetBox(session.BoxID)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, err
|
||||||
|
}
|
||||||
|
return s.resultForBox(box, ""), nil
|
||||||
|
}
|
||||||
|
if session.Status != ResumableStatusProcessing || session.BoxID == "" {
|
||||||
|
return UploadResult{}, fmt.Errorf("upload session is not processing")
|
||||||
|
}
|
||||||
|
box, err := s.GetBox(session.BoxID)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, err
|
||||||
|
}
|
||||||
|
staged, err := s.resumableIncomingFiles(session)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, err
|
||||||
|
}
|
||||||
|
if len(staged) != len(box.Files) {
|
||||||
|
return UploadResult{}, fmt.Errorf("processing file count mismatch")
|
||||||
|
}
|
||||||
|
backend, err := s.storage.Backend(box.StorageBackendID)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, err
|
||||||
|
}
|
||||||
|
for i, incoming := range staged {
|
||||||
|
source, err := incoming.Open()
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, err
|
||||||
|
}
|
||||||
|
file := box.Files[i]
|
||||||
|
if err := s.writeUploadedObject(ctx, backend, file.ObjectKey, source, incoming.Size(), 0, incoming.ContentType()); err != nil {
|
||||||
|
source.Close()
|
||||||
|
_ = backend.Delete(context.Background(), file.ObjectKey)
|
||||||
|
box.Files[i].ProcessingError = err.Error()
|
||||||
|
_ = s.saveBoxRecord(box)
|
||||||
|
return UploadResult{}, err
|
||||||
|
}
|
||||||
|
source.Close()
|
||||||
|
box.Files[i].Processing = false
|
||||||
|
box.Files[i].ProcessingError = ""
|
||||||
|
box.Files[i].UploadedAt = time.Now().UTC()
|
||||||
|
if err := s.saveBoxRecord(box); err != nil {
|
||||||
|
return UploadResult{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := s.writeBoxMetadata(box); err != nil {
|
||||||
|
s.logger.Warn("box metadata write failed after resumable processing", "source", "storage", "severity", "warn", "code", 4020, "box_id", box.ID, "error", err.Error())
|
||||||
|
}
|
||||||
|
if err := os.RemoveAll(s.resumableSessionDirFor(session)); err != nil {
|
||||||
|
return UploadResult{}, err
|
||||||
|
}
|
||||||
|
session.Status = ResumableStatusCompleted
|
||||||
|
session.UpdatedAt = time.Now().UTC()
|
||||||
|
if err := s.saveResumableSession(session); err != nil {
|
||||||
|
return UploadResult{}, err
|
||||||
|
}
|
||||||
|
return s.resultForBox(box, ""), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) CompleteUploadedResumableSession(ctx context.Context, sessionID string) (UploadResult, ResumableSession, error) {
|
||||||
|
session, err := s.GetResumableSession(sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
if err := resumableSessionWritable(session); err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
completeFiles := make([]ResumableFile, 0, len(session.Files))
|
||||||
|
for _, file := range session.Files {
|
||||||
|
if resumableFileComplete(file) {
|
||||||
|
completeFiles = append(completeFiles, file)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(completeFiles) == 0 {
|
||||||
|
return UploadResult{}, ResumableSession{}, fmt.Errorf("no fully uploaded files to finish")
|
||||||
|
}
|
||||||
|
partial := session
|
||||||
|
partial.Files = completeFiles
|
||||||
|
staged, err := s.resumableIncomingFiles(partial)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := s.CreateBoxFromIncomingContext(ctx, staged, session.Options)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
if err := os.RemoveAll(s.resumableSessionDirFor(session)); err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
session.Status = ResumableStatusCompleted
|
||||||
|
session.BoxID = result.BoxID
|
||||||
|
session.Files = completeFiles
|
||||||
|
session.UpdatedAt = time.Now().UTC()
|
||||||
|
if err := s.deleteResumableSession(session.ID); err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
return result, session, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (s *UploadService) CancelResumableSession(sessionID string) error {
|
func (s *UploadService) CancelResumableSession(sessionID string) error {
|
||||||
session, err := s.GetResumableSession(sessionID)
|
session, err := s.GetResumableSession(sessionID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
session.Status = ResumableStatusCancelled
|
if err := os.RemoveAll(s.resumableSessionDirFor(session)); err != nil {
|
||||||
session.UpdatedAt = time.Now().UTC()
|
|
||||||
if err := s.saveResumableSession(session); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return os.RemoveAll(s.resumableSessionDir(session.ID))
|
return s.deleteResumableSession(session.ID)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *UploadService) CleanupExpiredResumableSessions(now time.Time) (int, error) {
|
func (s *UploadService) CleanupExpiredResumableSessions(now time.Time) (int, error) {
|
||||||
@@ -256,7 +470,9 @@ func (s *UploadService) CleanupExpiredResumableSessions(now time.Time) (int, err
|
|||||||
if err := json.Unmarshal(value, &session); err != nil {
|
if err := json.Unmarshal(value, &session); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if !session.ExpiresAt.After(now) || session.Status != ResumableStatusUploading {
|
if session.Status == ResumableStatusCompleted ||
|
||||||
|
session.Status == ResumableStatusCancelled ||
|
||||||
|
(session.Status == ResumableStatusUploading && !session.ExpiresAt.After(now)) {
|
||||||
candidates = append(candidates, session)
|
candidates = append(candidates, session)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
@@ -266,7 +482,7 @@ func (s *UploadService) CleanupExpiredResumableSessions(now time.Time) (int, err
|
|||||||
return 0, err
|
return 0, err
|
||||||
}
|
}
|
||||||
for _, session := range candidates {
|
for _, session := range candidates {
|
||||||
if err := os.RemoveAll(s.resumableSessionDir(session.ID)); err != nil {
|
if err := os.RemoveAll(s.resumableSessionDirFor(session)); err != nil {
|
||||||
return 0, err
|
return 0, err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -285,6 +501,16 @@ func (s *UploadService) CleanupExpiredResumableSessions(now time.Time) (int, err
|
|||||||
return len(candidates), err
|
return len(candidates), err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) deleteResumableSession(sessionID string) error {
|
||||||
|
return s.db.Update(func(tx *bbolt.Tx) error {
|
||||||
|
bucket := tx.Bucket(resumableUploadsBucket)
|
||||||
|
if bucket == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return bucket.Delete([]byte(sessionID))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
func (s *UploadService) saveResumableSession(session ResumableSession) error {
|
func (s *UploadService) saveResumableSession(session ResumableSession) error {
|
||||||
if err := s.ensureResumableBucket(); err != nil {
|
if err := s.ensureResumableBucket(); err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -341,71 +567,102 @@ func resumableFileKey(name string, size int64, fingerprint string) string {
|
|||||||
return strings.TrimSpace(fingerprint) + "|" + filepath.Base(strings.TrimSpace(name)) + "|" + fmt.Sprintf("%d", size)
|
return strings.TrimSpace(fingerprint) + "|" + filepath.Base(strings.TrimSpace(name)) + "|" + fmt.Sprintf("%d", size)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *UploadService) assembleResumableFiles(ctx context.Context, session ResumableSession) ([]IncomingFile, func(), error) {
|
type resumableIncomingFile struct {
|
||||||
assembledDir := filepath.Join(s.resumableSessionDir(session.ID), "assembled")
|
service *UploadService
|
||||||
if err := os.MkdirAll(assembledDir, 0o755); err != nil {
|
session ResumableSession
|
||||||
return nil, func() {}, err
|
file ResumableFile
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f resumableIncomingFile) Name() string {
|
||||||
|
return f.file.Name
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f resumableIncomingFile) Size() int64 {
|
||||||
|
return f.file.Size
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f resumableIncomingFile) ContentType() string {
|
||||||
|
return f.file.ContentType
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f resumableIncomingFile) Open() (io.ReadCloser, error) {
|
||||||
|
return &resumableChunkReader{
|
||||||
|
service: f.service,
|
||||||
|
session: f.session,
|
||||||
|
file: f.file,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type resumableChunkReader struct {
|
||||||
|
service *UploadService
|
||||||
|
session ResumableSession
|
||||||
|
file ResumableFile
|
||||||
|
index int
|
||||||
|
current *os.File
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *resumableChunkReader) Read(p []byte) (int, error) {
|
||||||
|
for {
|
||||||
|
if r.current == nil {
|
||||||
|
if r.index >= r.file.ChunkCount {
|
||||||
|
return 0, io.EOF
|
||||||
}
|
}
|
||||||
cleanup := func() {
|
chunk, err := os.Open(r.service.resumableChunkPathFor(r.session, r.file.ID, r.index))
|
||||||
_ = os.RemoveAll(assembledDir)
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
}
|
}
|
||||||
|
r.current = chunk
|
||||||
|
}
|
||||||
|
n, err := r.current.Read(p)
|
||||||
|
if err == io.EOF {
|
||||||
|
if closeErr := r.current.Close(); closeErr != nil {
|
||||||
|
r.current = nil
|
||||||
|
return n, closeErr
|
||||||
|
}
|
||||||
|
r.current = nil
|
||||||
|
r.index++
|
||||||
|
if n > 0 {
|
||||||
|
return n, nil
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *resumableChunkReader) Close() error {
|
||||||
|
if r.current == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
err := r.current.Close()
|
||||||
|
r.current = nil
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) resumableIncomingFiles(session ResumableSession) ([]IncomingFile, error) {
|
||||||
staged := make([]IncomingFile, 0, len(session.Files))
|
staged := make([]IncomingFile, 0, len(session.Files))
|
||||||
for _, file := range session.Files {
|
for _, file := range session.Files {
|
||||||
if len(file.UploadedChunks) != file.ChunkCount {
|
if len(file.UploadedChunks) != file.ChunkCount {
|
||||||
cleanup()
|
return nil, fmt.Errorf("file %s is missing chunks", file.Name)
|
||||||
return nil, func() {}, fmt.Errorf("file %s is missing chunks", file.Name)
|
|
||||||
}
|
|
||||||
assembledPath := filepath.Join(assembledDir, file.ID)
|
|
||||||
target, err := os.OpenFile(assembledPath, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
|
|
||||||
if err != nil {
|
|
||||||
cleanup()
|
|
||||||
return nil, func() {}, err
|
|
||||||
}
|
}
|
||||||
var written int64
|
var written int64
|
||||||
for i := 0; i < file.ChunkCount; i++ {
|
for i := 0; i < file.ChunkCount; i++ {
|
||||||
select {
|
info, err := os.Stat(s.resumableChunkPathFor(session, file.ID, i))
|
||||||
case <-ctx.Done():
|
|
||||||
_ = target.Close()
|
|
||||||
cleanup()
|
|
||||||
return nil, func() {}, ctx.Err()
|
|
||||||
default:
|
|
||||||
}
|
|
||||||
chunk, err := os.Open(s.resumableChunkPath(session.ID, file.ID, i))
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = target.Close()
|
return nil, fmt.Errorf("file %s is missing chunks", file.Name)
|
||||||
cleanup()
|
|
||||||
return nil, func() {}, fmt.Errorf("file %s is missing chunks", file.Name)
|
|
||||||
}
|
}
|
||||||
n, copyErr := io.Copy(target, chunk)
|
written += info.Size()
|
||||||
closeErr := chunk.Close()
|
|
||||||
if copyErr != nil {
|
|
||||||
_ = target.Close()
|
|
||||||
cleanup()
|
|
||||||
return nil, func() {}, copyErr
|
|
||||||
}
|
|
||||||
if closeErr != nil {
|
|
||||||
_ = target.Close()
|
|
||||||
cleanup()
|
|
||||||
return nil, func() {}, closeErr
|
|
||||||
}
|
|
||||||
written += n
|
|
||||||
}
|
|
||||||
if err := target.Close(); err != nil {
|
|
||||||
cleanup()
|
|
||||||
return nil, func() {}, err
|
|
||||||
}
|
}
|
||||||
if written != file.Size {
|
if written != file.Size {
|
||||||
cleanup()
|
return nil, fmt.Errorf("chunk size total mismatch")
|
||||||
return nil, func() {}, fmt.Errorf("assembled file size mismatch")
|
|
||||||
}
|
}
|
||||||
staged = append(staged, StagedUploadFile{
|
staged = append(staged, resumableIncomingFile{
|
||||||
Filename: file.Name,
|
service: s,
|
||||||
FileSize: file.Size,
|
session: session,
|
||||||
MIMEType: file.ContentType,
|
file: file,
|
||||||
Path: assembledPath,
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
return staged, cleanup, nil
|
return staged, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func resumableSessionWritable(session ResumableSession) error {
|
func resumableSessionWritable(session ResumableSession) error {
|
||||||
@@ -418,6 +675,10 @@ func resumableSessionWritable(session ResumableSession) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func resumableFileComplete(file ResumableFile) bool {
|
||||||
|
return file.ChunkCount > 0 && len(file.UploadedChunks) == file.ChunkCount
|
||||||
|
}
|
||||||
|
|
||||||
func expectedChunkSize(fileSize, chunkSize int64, index int) int64 {
|
func expectedChunkSize(fileSize, chunkSize int64, index int) int64 {
|
||||||
offset := int64(index) * chunkSize
|
offset := int64(index) * chunkSize
|
||||||
remaining := fileSize - offset
|
remaining := fileSize - offset
|
||||||
@@ -441,14 +702,34 @@ func addChunkIndex(chunks []int, index int) []int {
|
|||||||
return chunks
|
return chunks
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func resumableTokenHash(sessionID, token string) string {
|
||||||
|
sum := sha256.Sum256([]byte("warpbox-resumable:" + sessionID + ":" + token))
|
||||||
|
return hex.EncodeToString(sum[:])
|
||||||
|
}
|
||||||
|
|
||||||
func (s *UploadService) resumableSessionDir(sessionID string) string {
|
func (s *UploadService) resumableSessionDir(sessionID string) string {
|
||||||
return filepath.Join(s.dataDir, "tmp", "uploads", sessionID)
|
return filepath.Join(s.dataDir, "tmp", "uploads", sessionID)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) resumableSessionDirFor(session ResumableSession) string {
|
||||||
|
if strings.TrimSpace(session.ChunkRoot) != "" {
|
||||||
|
return filepath.Join(session.ChunkRoot, session.ID)
|
||||||
|
}
|
||||||
|
return s.resumableSessionDir(session.ID)
|
||||||
|
}
|
||||||
|
|
||||||
func (s *UploadService) resumableFileDir(sessionID, fileID string) string {
|
func (s *UploadService) resumableFileDir(sessionID, fileID string) string {
|
||||||
return filepath.Join(s.resumableSessionDir(sessionID), fileID)
|
return filepath.Join(s.resumableSessionDir(sessionID), fileID)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) resumableFileDirFor(session ResumableSession, fileID string) string {
|
||||||
|
return filepath.Join(s.resumableSessionDirFor(session), fileID)
|
||||||
|
}
|
||||||
|
|
||||||
func (s *UploadService) resumableChunkPath(sessionID, fileID string, index int) string {
|
func (s *UploadService) resumableChunkPath(sessionID, fileID string, index int) string {
|
||||||
return filepath.Join(s.resumableFileDir(sessionID, fileID), fmt.Sprintf("%06d.part", index))
|
return filepath.Join(s.resumableFileDir(sessionID, fileID), fmt.Sprintf("%06d.part", index))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) resumableChunkPathFor(session ResumableSession, fileID string, index int) string {
|
||||||
|
return filepath.Join(s.resumableFileDirFor(session, fileID), fmt.Sprintf("%06d.part", index))
|
||||||
|
}
|
||||||
|
|||||||
@@ -37,6 +37,11 @@ type UploadPolicySettings struct {
|
|||||||
ShortWindowSeconds int `json:"shortWindowSeconds"`
|
ShortWindowSeconds int `json:"shortWindowSeconds"`
|
||||||
AnonymousStorageBackend string `json:"anonymousStorageBackend"`
|
AnonymousStorageBackend string `json:"anonymousStorageBackend"`
|
||||||
UserStorageBackend string `json:"userStorageBackend"`
|
UserStorageBackend string `json:"userStorageBackend"`
|
||||||
|
ResumableUploadsEnabled bool `json:"resumableUploadsEnabled"`
|
||||||
|
ResumableChunkSizeMB float64 `json:"resumableChunkSizeMb"`
|
||||||
|
ResumableRetentionHours int `json:"resumableRetentionHours"`
|
||||||
|
ResumableChunkMode string `json:"resumableChunkMode"`
|
||||||
|
ResumableChunkPath string `json:"resumableChunkPath"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type UsageRecord struct {
|
type UsageRecord struct {
|
||||||
@@ -89,6 +94,11 @@ func NewSettingsService(db *bbolt.DB, defaults config.SettingsDefaults) (*Settin
|
|||||||
ShortWindowSeconds: defaults.ShortWindowSeconds,
|
ShortWindowSeconds: defaults.ShortWindowSeconds,
|
||||||
AnonymousStorageBackend: defaults.AnonymousStorageBackend,
|
AnonymousStorageBackend: defaults.AnonymousStorageBackend,
|
||||||
UserStorageBackend: defaults.UserStorageBackend,
|
UserStorageBackend: defaults.UserStorageBackend,
|
||||||
|
ResumableUploadsEnabled: defaults.ResumableUploadsEnabled,
|
||||||
|
ResumableChunkSizeMB: defaults.ResumableChunkSizeMB,
|
||||||
|
ResumableRetentionHours: defaults.ResumableRetentionHours,
|
||||||
|
ResumableChunkMode: defaults.ResumableChunkMode,
|
||||||
|
ResumableChunkPath: defaults.ResumableChunkPath,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
service.defaults = service.withBuiltinDefaultGaps(service.defaults)
|
service.defaults = service.withBuiltinDefaultGaps(service.defaults)
|
||||||
@@ -143,6 +153,15 @@ func (s *SettingsService) withBuiltinDefaultGaps(settings UploadPolicySettings)
|
|||||||
if strings.TrimSpace(settings.UserStorageBackend) == "" {
|
if strings.TrimSpace(settings.UserStorageBackend) == "" {
|
||||||
settings.UserStorageBackend = StorageBackendLocal
|
settings.UserStorageBackend = StorageBackendLocal
|
||||||
}
|
}
|
||||||
|
if settings.ResumableChunkSizeMB <= 0 {
|
||||||
|
settings.ResumableChunkSizeMB = 8
|
||||||
|
}
|
||||||
|
if settings.ResumableRetentionHours <= 0 {
|
||||||
|
settings.ResumableRetentionHours = 24
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(settings.ResumableChunkMode) == "" {
|
||||||
|
settings.ResumableChunkMode = "same"
|
||||||
|
}
|
||||||
return settings
|
return settings
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -156,6 +175,13 @@ func (s *SettingsService) UploadPolicy() (UploadPolicySettings, error) {
|
|||||||
if err := json.Unmarshal(data, &settings); err != nil {
|
if err := json.Unmarshal(data, &settings); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
var raw map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(data, &raw); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if _, ok := raw["resumableUploadsEnabled"]; !ok {
|
||||||
|
settings.ResumableUploadsEnabled = s.defaults.ResumableUploadsEnabled
|
||||||
|
}
|
||||||
settings = s.withDefaultGaps(settings)
|
settings = s.withDefaultGaps(settings)
|
||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
@@ -217,6 +243,15 @@ func (s *SettingsService) withDefaultGaps(settings UploadPolicySettings) UploadP
|
|||||||
if strings.TrimSpace(settings.UserStorageBackend) == "" {
|
if strings.TrimSpace(settings.UserStorageBackend) == "" {
|
||||||
settings.UserStorageBackend = s.defaults.UserStorageBackend
|
settings.UserStorageBackend = s.defaults.UserStorageBackend
|
||||||
}
|
}
|
||||||
|
if settings.ResumableChunkSizeMB <= 0 {
|
||||||
|
settings.ResumableChunkSizeMB = s.defaults.ResumableChunkSizeMB
|
||||||
|
}
|
||||||
|
if settings.ResumableRetentionHours <= 0 {
|
||||||
|
settings.ResumableRetentionHours = s.defaults.ResumableRetentionHours
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(settings.ResumableChunkMode) == "" {
|
||||||
|
settings.ResumableChunkMode = s.defaults.ResumableChunkMode
|
||||||
|
}
|
||||||
return settings
|
return settings
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -422,6 +457,18 @@ func (s *SettingsService) validate(settings UploadPolicySettings) error {
|
|||||||
if settings.ShortWindowRequests <= 0 || settings.ShortWindowSeconds <= 0 {
|
if settings.ShortWindowRequests <= 0 || settings.ShortWindowSeconds <= 0 {
|
||||||
return fmt.Errorf("short-window rate limits must be positive")
|
return fmt.Errorf("short-window rate limits must be positive")
|
||||||
}
|
}
|
||||||
|
if settings.ResumableChunkSizeMB <= 0 {
|
||||||
|
return fmt.Errorf("resumable chunk size must be positive")
|
||||||
|
}
|
||||||
|
if settings.ResumableRetentionHours <= 0 {
|
||||||
|
return fmt.Errorf("resumable retention must be positive")
|
||||||
|
}
|
||||||
|
if settings.ResumableChunkMode != "same" && settings.ResumableChunkMode != "custom" {
|
||||||
|
return fmt.Errorf("resumable chunk storage mode is invalid")
|
||||||
|
}
|
||||||
|
if settings.ResumableChunkMode == "custom" && strings.TrimSpace(settings.ResumableChunkPath) == "" {
|
||||||
|
return fmt.Errorf("custom resumable chunk path is required")
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -130,6 +130,8 @@ type File struct {
|
|||||||
Thumbnail string `json:"thumbnail,omitempty"`
|
Thumbnail string `json:"thumbnail,omitempty"`
|
||||||
ObjectKey string `json:"objectKey,omitempty"`
|
ObjectKey string `json:"objectKey,omitempty"`
|
||||||
ThumbnailObjectKey string `json:"thumbnailObjectKey,omitempty"`
|
ThumbnailObjectKey string `json:"thumbnailObjectKey,omitempty"`
|
||||||
|
Processing bool `json:"processing,omitempty"`
|
||||||
|
ProcessingError string `json:"processingError,omitempty"`
|
||||||
UploadedAt time.Time `json:"uploadedAt"`
|
UploadedAt time.Time `json:"uploadedAt"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -150,6 +152,7 @@ type ResultFile struct {
|
|||||||
Size string `json:"size"`
|
Size string `json:"size"`
|
||||||
URL string `json:"url"`
|
URL string `json:"url"`
|
||||||
ThumbnailURL string `json:"thumbnailUrl"`
|
ThumbnailURL string `json:"thumbnailUrl"`
|
||||||
|
Processing bool `json:"processing,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type AdminStats struct {
|
type AdminStats struct {
|
||||||
@@ -254,6 +257,10 @@ func (s *UploadService) CreateBox(files []*multipart.FileHeader, opts UploadOpti
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *UploadService) CreateBoxFromIncoming(files []IncomingFile, opts UploadOptions) (UploadResult, error) {
|
func (s *UploadService) CreateBoxFromIncoming(files []IncomingFile, opts UploadOptions) (UploadResult, error) {
|
||||||
|
return s.CreateBoxFromIncomingContext(context.Background(), files, opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) CreateBoxFromIncomingContext(ctx context.Context, files []IncomingFile, opts UploadOptions) (UploadResult, error) {
|
||||||
if len(files) == 0 {
|
if len(files) == 0 {
|
||||||
return UploadResult{}, fmt.Errorf("no files were uploaded")
|
return UploadResult{}, fmt.Errorf("no files were uploaded")
|
||||||
}
|
}
|
||||||
@@ -297,7 +304,7 @@ func (s *UploadService) CreateBoxFromIncoming(files []IncomingFile, opts UploadO
|
|||||||
box.PasswordHash = hash
|
box.PasswordHash = hash
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := s.writeIncomingFilesToBox(&box, files, opts); err != nil {
|
if err := s.writeIncomingFilesToBox(ctx, &box, files, opts); err != nil {
|
||||||
return UploadResult{}, err
|
return UploadResult{}, err
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -331,7 +338,7 @@ func (s *UploadService) AppendIncomingFiles(boxID string, files []IncomingFile,
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return UploadResult{}, err
|
return UploadResult{}, err
|
||||||
}
|
}
|
||||||
if err := s.writeIncomingFilesToBox(&box, files, opts); err != nil {
|
if err := s.writeIncomingFilesToBox(context.Background(), &box, files, opts); err != nil {
|
||||||
return UploadResult{}, err
|
return UploadResult{}, err
|
||||||
}
|
}
|
||||||
if err := s.SaveBox(box); err != nil {
|
if err := s.SaveBox(box); err != nil {
|
||||||
@@ -352,7 +359,7 @@ func (s *UploadService) AppendIncomingFiles(boxID string, files []IncomingFile,
|
|||||||
// appends the file metadata to box.Files. The box's StorageBackendID determines
|
// appends the file metadata to box.Files. The box's StorageBackendID determines
|
||||||
// where files land, so it works for both new and existing boxes.
|
// where files land, so it works for both new and existing boxes.
|
||||||
func (s *UploadService) writeFilesToBox(box *Box, files []*multipart.FileHeader, opts UploadOptions) error {
|
func (s *UploadService) writeFilesToBox(box *Box, files []*multipart.FileHeader, opts UploadOptions) error {
|
||||||
return s.writeIncomingFilesToBox(box, multipartIncomingFiles(files), opts)
|
return s.writeIncomingFilesToBox(context.Background(), box, multipartIncomingFiles(files), opts)
|
||||||
}
|
}
|
||||||
|
|
||||||
func multipartIncomingFiles(files []*multipart.FileHeader) []IncomingFile {
|
func multipartIncomingFiles(files []*multipart.FileHeader) []IncomingFile {
|
||||||
@@ -363,7 +370,7 @@ func multipartIncomingFiles(files []*multipart.FileHeader) []IncomingFile {
|
|||||||
return incoming
|
return incoming
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *UploadService) writeIncomingFilesToBox(box *Box, files []IncomingFile, opts UploadOptions) error {
|
func (s *UploadService) writeIncomingFilesToBox(ctx context.Context, box *Box, files []IncomingFile, opts UploadOptions) error {
|
||||||
backend, err := s.storage.Backend(box.StorageBackendID)
|
backend, err := s.storage.Backend(box.StorageBackendID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -399,8 +406,9 @@ func (s *UploadService) writeIncomingFilesToBox(box *Box, files []IncomingFile,
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := s.writeUploadedObject(context.Background(), backend, objectKey, file, incoming.Size(), maxSize, contentType); err != nil {
|
if err := s.writeUploadedObject(ctx, backend, objectKey, file, incoming.Size(), maxSize, contentType); err != nil {
|
||||||
file.Close()
|
file.Close()
|
||||||
|
_ = backend.Delete(context.Background(), objectKey)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
file.Close()
|
file.Close()
|
||||||
@@ -811,6 +819,9 @@ func (s *UploadService) ThumbnailObjectKey(box Box, file File) string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *UploadService) OpenFileObject(ctx context.Context, box Box, file File) (StorageObject, error) {
|
func (s *UploadService) OpenFileObject(ctx context.Context, box Box, file File) (StorageObject, error) {
|
||||||
|
if file.Processing {
|
||||||
|
return StorageObject{}, fmt.Errorf("file is still processing")
|
||||||
|
}
|
||||||
backend, err := s.storage.Backend(s.BoxStorageBackendID(box))
|
backend, err := s.storage.Backend(s.BoxStorageBackendID(box))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return StorageObject{}, err
|
return StorageObject{}, err
|
||||||
@@ -932,6 +943,13 @@ func (s *UploadService) WriteZip(w io.Writer, box Box) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *UploadService) SaveBox(box Box) error {
|
func (s *UploadService) SaveBox(box Box) error {
|
||||||
|
if err := s.saveBoxRecord(box); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return s.writeBoxMetadata(box)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) saveBoxRecord(box Box) error {
|
||||||
if box.StorageBackendID == "" {
|
if box.StorageBackendID == "" {
|
||||||
box.StorageBackendID = StorageBackendLocal
|
box.StorageBackendID = StorageBackendLocal
|
||||||
}
|
}
|
||||||
@@ -941,10 +959,7 @@ func (s *UploadService) SaveBox(box Box) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
return s.db.Update(func(tx *bbolt.Tx) error {
|
return s.db.Update(func(tx *bbolt.Tx) error {
|
||||||
if err := tx.Bucket(boxesBucket).Put([]byte(box.ID), data); err != nil {
|
return tx.Bucket(boxesBucket).Put([]byte(box.ID), data)
|
||||||
return err
|
|
||||||
}
|
|
||||||
return s.writeBoxMetadata(box)
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -957,6 +972,7 @@ func (s *UploadService) resultForBox(box Box, deleteToken string) UploadResult {
|
|||||||
Size: helpers.FormatBytes(file.Size),
|
Size: helpers.FormatBytes(file.Size),
|
||||||
URL: fmt.Sprintf("%s/d/%s/f/%s", s.baseURL, box.ID, file.ID),
|
URL: fmt.Sprintf("%s/d/%s/f/%s", s.baseURL, box.ID, file.ID),
|
||||||
ThumbnailURL: fmt.Sprintf("%s/d/%s/thumb/%s", s.baseURL, box.ID, file.ID),
|
ThumbnailURL: fmt.Sprintf("%s/d/%s/thumb/%s", s.baseURL, box.ID, file.ID),
|
||||||
|
Processing: file.Processing,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1016,9 +1032,26 @@ func (s *UploadService) writeUploadedObject(ctx context.Context, backend Storage
|
|||||||
reader = io.LimitReader(source, maxSize)
|
reader = io.LimitReader(source, maxSize)
|
||||||
putSize = size
|
putSize = size
|
||||||
}
|
}
|
||||||
|
if ctx != nil {
|
||||||
|
reader = contextReader{ctx: ctx, reader: reader}
|
||||||
|
}
|
||||||
return backend.Put(ctx, key, reader, putSize, contentType)
|
return backend.Put(ctx, key, reader, putSize, contentType)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type contextReader struct {
|
||||||
|
ctx context.Context
|
||||||
|
reader io.Reader
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r contextReader) Read(p []byte) (int, error) {
|
||||||
|
select {
|
||||||
|
case <-r.ctx.Done():
|
||||||
|
return 0, r.ctx.Err()
|
||||||
|
default:
|
||||||
|
return r.reader.Read(p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func boxObjectKey(boxID, name string) string {
|
func boxObjectKey(boxID, name string) string {
|
||||||
return filepath.ToSlash(filepath.Join(boxID, name))
|
return filepath.ToSlash(filepath.Join(boxID, name))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -133,10 +133,32 @@ func TestResumableSessionUploadOutOfOrderAndComplete(t *testing.T) {
|
|||||||
Size: 11,
|
Size: 11,
|
||||||
ContentType: "text/plain",
|
ContentType: "text/plain",
|
||||||
Fingerprint: "sha256:first-chunk",
|
Fingerprint: "sha256:first-chunk",
|
||||||
}}, UploadOptions{MaxDays: 1, Password: "secret"}, 4, time.Hour)
|
}}, UploadOptions{MaxDays: 1, Password: "secret"}, 4, time.Hour, "")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("CreateResumableSession returned error: %v", err)
|
t.Fatalf("CreateResumableSession returned error: %v", err)
|
||||||
}
|
}
|
||||||
|
if session.ResumeToken == "" || session.ResumeTokenHash == "" {
|
||||||
|
t.Fatalf("resumable session did not create resume token: %+v", session)
|
||||||
|
}
|
||||||
|
if !service.VerifyResumableToken(session, session.ResumeToken) {
|
||||||
|
t.Fatalf("VerifyResumableToken rejected correct token")
|
||||||
|
}
|
||||||
|
if service.VerifyResumableToken(session, "wrong-token") {
|
||||||
|
t.Fatalf("VerifyResumableToken accepted wrong token")
|
||||||
|
}
|
||||||
|
stored, err := service.GetResumableSession(session.ID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetResumableSession returned error: %v", err)
|
||||||
|
}
|
||||||
|
if stored.ResumeToken != "" {
|
||||||
|
t.Fatalf("stored session leaked raw resume token")
|
||||||
|
}
|
||||||
|
if strings.Contains(stored.ResumeTokenHash, session.ResumeToken) {
|
||||||
|
t.Fatalf("stored token hash contains raw token")
|
||||||
|
}
|
||||||
|
if !service.VerifyResumableToken(stored, session.ResumeToken) {
|
||||||
|
t.Fatalf("stored session rejected correct token")
|
||||||
|
}
|
||||||
if session.Options.Password != "" || session.Options.PasswordHash == "" || session.Options.PasswordSalt == "" {
|
if session.Options.Password != "" || session.Options.PasswordHash == "" || session.Options.PasswordSalt == "" {
|
||||||
t.Fatalf("resumable session did not hash password before storage: %+v", session.Options)
|
t.Fatalf("resumable session did not hash password before storage: %+v", session.Options)
|
||||||
}
|
}
|
||||||
@@ -181,6 +203,13 @@ func TestResumableSessionUploadOutOfOrderAndComplete(t *testing.T) {
|
|||||||
if _, err := os.Stat(service.resumableSessionDir(session.ID)); !os.IsNotExist(err) {
|
if _, err := os.Stat(service.resumableSessionDir(session.ID)); !os.IsNotExist(err) {
|
||||||
t.Fatalf("resumable temp dir after complete error = %v, want os.ErrNotExist", err)
|
t.Fatalf("resumable temp dir after complete error = %v, want os.ErrNotExist", err)
|
||||||
}
|
}
|
||||||
|
replayed, replayedSession, err := service.CompleteResumableSession(testContext(), session.ID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CompleteResumableSession replay returned error: %v", err)
|
||||||
|
}
|
||||||
|
if replayed.BoxID != result.BoxID || replayedSession.Status != ResumableStatusCompleted {
|
||||||
|
t.Fatalf("replayed result = %+v, session = %+v, want box %s completed", replayed, replayedSession, result.BoxID)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResumableCompleteRejectsMissingChunks(t *testing.T) {
|
func TestResumableCompleteRejectsMissingChunks(t *testing.T) {
|
||||||
@@ -189,7 +218,7 @@ func TestResumableCompleteRejectsMissingChunks(t *testing.T) {
|
|||||||
Name: "note.txt",
|
Name: "note.txt",
|
||||||
Size: 8,
|
Size: 8,
|
||||||
ContentType: "text/plain",
|
ContentType: "text/plain",
|
||||||
}}, UploadOptions{MaxDays: 1}, 4, time.Hour)
|
}}, UploadOptions{MaxDays: 1}, 4, time.Hour, "")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("CreateResumableSession returned error: %v", err)
|
t.Fatalf("CreateResumableSession returned error: %v", err)
|
||||||
}
|
}
|
||||||
@@ -201,6 +230,73 @@ func TestResumableCompleteRejectsMissingChunks(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestResumablePartialCompleteKeepsOnlyFinishedFiles(t *testing.T) {
|
||||||
|
service := newTestUploadService(t)
|
||||||
|
session, err := service.CreateResumableSession([]ResumableFileInput{
|
||||||
|
{Name: "done.txt", Size: 4, ContentType: "text/plain", Fingerprint: "done"},
|
||||||
|
{Name: "partial.txt", Size: 8, ContentType: "text/plain", Fingerprint: "partial"},
|
||||||
|
}, UploadOptions{MaxDays: 1}, 4, time.Hour, "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CreateResumableSession returned error: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := service.PutResumableChunk(testContext(), session.ID, session.Files[0].ID, 0, strings.NewReader("done")); err != nil {
|
||||||
|
t.Fatalf("PutResumableChunk done returned error: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := service.PutResumableChunk(testContext(), session.ID, session.Files[1].ID, 0, strings.NewReader("part")); err != nil {
|
||||||
|
t.Fatalf("PutResumableChunk partial returned error: %v", err)
|
||||||
|
}
|
||||||
|
result, completed, err := service.CompleteUploadedResumableSession(testContext(), session.ID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CompleteUploadedResumableSession returned error: %v", err)
|
||||||
|
}
|
||||||
|
if completed.Status != ResumableStatusCompleted || completed.BoxID != result.BoxID || len(completed.Files) != 1 {
|
||||||
|
t.Fatalf("completed session = %+v, result = %+v", completed, result)
|
||||||
|
}
|
||||||
|
box := getTestBox(t, service, result.BoxID)
|
||||||
|
if len(box.Files) != 1 || box.Files[0].Name != "done.txt" {
|
||||||
|
t.Fatalf("partial completion box files = %+v", box.Files)
|
||||||
|
}
|
||||||
|
object, err := service.OpenFileObject(testContext(), box, box.Files[0])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("OpenFileObject returned error: %v", err)
|
||||||
|
}
|
||||||
|
data, err := io.ReadAll(object.Body)
|
||||||
|
object.Body.Close()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadAll returned error: %v", err)
|
||||||
|
}
|
||||||
|
if string(data) != "done" {
|
||||||
|
t.Fatalf("partial completion object = %q", string(data))
|
||||||
|
}
|
||||||
|
if _, err := service.GetResumableSession(session.ID); !os.IsNotExist(err) {
|
||||||
|
t.Fatalf("GetResumableSession after partial complete error = %v, want os.ErrNotExist", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(service.resumableSessionDir(session.ID)); !os.IsNotExist(err) {
|
||||||
|
t.Fatalf("resumable temp dir after partial complete error = %v, want os.ErrNotExist", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResumablePartialCompleteRejectsNoFinishedFiles(t *testing.T) {
|
||||||
|
service := newTestUploadService(t)
|
||||||
|
session, err := service.CreateResumableSession([]ResumableFileInput{{
|
||||||
|
Name: "partial.txt",
|
||||||
|
Size: 8,
|
||||||
|
ContentType: "text/plain",
|
||||||
|
}}, UploadOptions{MaxDays: 1}, 4, time.Hour, "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CreateResumableSession returned error: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := service.PutResumableChunk(testContext(), session.ID, session.Files[0].ID, 0, strings.NewReader("part")); err != nil {
|
||||||
|
t.Fatalf("PutResumableChunk returned error: %v", err)
|
||||||
|
}
|
||||||
|
if _, _, err := service.CompleteUploadedResumableSession(testContext(), session.ID); err == nil {
|
||||||
|
t.Fatalf("CompleteUploadedResumableSession accepted no completed files")
|
||||||
|
}
|
||||||
|
if _, err := service.GetResumableSession(session.ID); err != nil {
|
||||||
|
t.Fatalf("GetResumableSession after failed partial complete returned error: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResumableSessionCanAddFilesBeforeComplete(t *testing.T) {
|
func TestResumableSessionCanAddFilesBeforeComplete(t *testing.T) {
|
||||||
service := newTestUploadService(t)
|
service := newTestUploadService(t)
|
||||||
session, err := service.CreateResumableSession([]ResumableFileInput{{
|
session, err := service.CreateResumableSession([]ResumableFileInput{{
|
||||||
@@ -208,7 +304,7 @@ func TestResumableSessionCanAddFilesBeforeComplete(t *testing.T) {
|
|||||||
Size: 4,
|
Size: 4,
|
||||||
ContentType: "text/plain",
|
ContentType: "text/plain",
|
||||||
Fingerprint: "one",
|
Fingerprint: "one",
|
||||||
}}, UploadOptions{MaxDays: 1}, 4, time.Hour)
|
}}, UploadOptions{MaxDays: 1}, 4, time.Hour, "")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("CreateResumableSession returned error: %v", err)
|
t.Fatalf("CreateResumableSession returned error: %v", err)
|
||||||
}
|
}
|
||||||
@@ -264,7 +360,7 @@ func TestResumableCleanupRemovesExpiredSessionsAndChunks(t *testing.T) {
|
|||||||
Name: "note.txt",
|
Name: "note.txt",
|
||||||
Size: 4,
|
Size: 4,
|
||||||
ContentType: "text/plain",
|
ContentType: "text/plain",
|
||||||
}}, UploadOptions{MaxDays: 1}, 4, time.Millisecond)
|
}}, UploadOptions{MaxDays: 1}, 4, time.Millisecond, "")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("CreateResumableSession returned error: %v", err)
|
t.Fatalf("CreateResumableSession returned error: %v", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -48,6 +48,10 @@
|
|||||||
width: 100%;
|
width: 100%;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.upload-options .form-footer .upload-new-button {
|
||||||
|
margin-top: -0.25rem;
|
||||||
|
}
|
||||||
|
|
||||||
.hero-copy {
|
.hero-copy {
|
||||||
text-align: center;
|
text-align: center;
|
||||||
}
|
}
|
||||||
@@ -371,6 +375,62 @@ button {
|
|||||||
background: var(--surface-1-hover);
|
background: var(--surface-1-hover);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.upload-file-waiting {
|
||||||
|
border-color: color-mix(in srgb, var(--primary) 42%, var(--border));
|
||||||
|
background: color-mix(in srgb, var(--primary) 8%, var(--background));
|
||||||
|
}
|
||||||
|
|
||||||
|
.upload-file-complete .file-progress span {
|
||||||
|
background: var(--success);
|
||||||
|
}
|
||||||
|
|
||||||
|
.upload-file-state {
|
||||||
|
grid-column: 1 / -1;
|
||||||
|
color: var(--muted-foreground);
|
||||||
|
font-size: 0.72rem;
|
||||||
|
text-align: right;
|
||||||
|
}
|
||||||
|
|
||||||
|
.upload-recovery-overlay {
|
||||||
|
position: fixed;
|
||||||
|
inset: 0;
|
||||||
|
z-index: 80;
|
||||||
|
display: grid;
|
||||||
|
place-items: center;
|
||||||
|
padding: 1rem;
|
||||||
|
background: color-mix(in srgb, var(--background) 72%, transparent);
|
||||||
|
backdrop-filter: blur(10px);
|
||||||
|
}
|
||||||
|
|
||||||
|
.upload-recovery-modal {
|
||||||
|
width: min(34rem, 100%);
|
||||||
|
box-shadow: var(--shadow-lg);
|
||||||
|
}
|
||||||
|
|
||||||
|
.upload-recovery-modal h2 {
|
||||||
|
margin: 0 0 0.65rem;
|
||||||
|
font-size: 1.35rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.upload-recovery-modal p {
|
||||||
|
margin: 0;
|
||||||
|
color: var(--muted-foreground);
|
||||||
|
line-height: 1.55;
|
||||||
|
}
|
||||||
|
|
||||||
|
.upload-recovery-actions {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||||
|
gap: 0.7rem;
|
||||||
|
margin-top: 1.25rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (max-width: 640px) {
|
||||||
|
.upload-recovery-actions {
|
||||||
|
grid-template-columns: 1fr;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
.result-item small,
|
.result-item small,
|
||||||
.download-item small,
|
.download-item small,
|
||||||
.result-item code,
|
.result-item code,
|
||||||
|
|||||||
@@ -46,6 +46,15 @@
|
|||||||
text-decoration: none;
|
text-decoration: none;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.upload-processing-alert {
|
||||||
|
margin: 1rem 0;
|
||||||
|
padding: .85rem 1rem;
|
||||||
|
border: 1px solid color-mix(in srgb, var(--primary) 45%, transparent);
|
||||||
|
border-radius: var(--radius);
|
||||||
|
background: color-mix(in srgb, var(--primary) 12%, transparent);
|
||||||
|
color: var(--foreground);
|
||||||
|
}
|
||||||
|
|
||||||
.thumb-link {
|
.thumb-link {
|
||||||
flex: 0 0 4.75rem;
|
flex: 0 0 4.75rem;
|
||||||
width: 4.75rem;
|
width: 4.75rem;
|
||||||
@@ -200,6 +209,15 @@
|
|||||||
padding: 0;
|
padding: 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.file-card.is-processing {
|
||||||
|
opacity: .62;
|
||||||
|
filter: grayscale(.25);
|
||||||
|
}
|
||||||
|
|
||||||
|
.file-card.is-processing .file-open {
|
||||||
|
cursor: wait;
|
||||||
|
}
|
||||||
|
|
||||||
.file-reaction-dock {
|
.file-reaction-dock {
|
||||||
position: static;
|
position: static;
|
||||||
z-index: 2;
|
z-index: 2;
|
||||||
|
|||||||
@@ -5,6 +5,17 @@
|
|||||||
window.open(url, "_blank", "noopener,noreferrer");
|
window.open(url, "_blank", "noopener,noreferrer");
|
||||||
};
|
};
|
||||||
|
|
||||||
|
window.Warpbox.absoluteURL = function absoluteURL(url) {
|
||||||
|
if (!url) {
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
return new URL(url, window.location.origin).href;
|
||||||
|
} catch (_) {
|
||||||
|
return url;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
window.Warpbox.writeClipboard = async function writeClipboard(text) {
|
window.Warpbox.writeClipboard = async function writeClipboard(text) {
|
||||||
if (navigator.clipboard && window.isSecureContext) {
|
if (navigator.clipboard && window.isSecureContext) {
|
||||||
await navigator.clipboard.writeText(text);
|
await navigator.clipboard.writeText(text);
|
||||||
@@ -26,6 +37,9 @@
|
|||||||
if (!text || !button) {
|
if (!text || !button) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
if (typeof text === "string" && (text.startsWith("/") || /^https?:\/\//i.test(text))) {
|
||||||
|
text = window.Warpbox.absoluteURL(text);
|
||||||
|
}
|
||||||
await window.Warpbox.writeClipboard(text);
|
await window.Warpbox.writeClipboard(text);
|
||||||
const previous = button.textContent;
|
const previous = button.textContent;
|
||||||
button.textContent = copiedLabel;
|
button.textContent = copiedLabel;
|
||||||
|
|||||||
@@ -124,7 +124,7 @@
|
|||||||
}
|
}
|
||||||
|
|
||||||
async function copyPreviewLink(button) {
|
async function copyPreviewLink(button) {
|
||||||
await window.Warpbox.writeClipboard(button.href);
|
await window.Warpbox.writeClipboard(window.Warpbox.absoluteURL(button.href));
|
||||||
const label = button.querySelector("[data-preview-label]");
|
const label = button.querySelector("[data-preview-label]");
|
||||||
if (!label) {
|
if (!label) {
|
||||||
return;
|
return;
|
||||||
@@ -164,11 +164,11 @@
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
if (action === "copy-preview") {
|
if (action === "copy-preview") {
|
||||||
await window.Warpbox.writeClipboard(file.previewURL);
|
await window.Warpbox.writeClipboard(window.Warpbox.absoluteURL(file.previewURL));
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
if (action === "copy-download") {
|
if (action === "copy-download") {
|
||||||
await window.Warpbox.writeClipboard(file.downloadURL);
|
await window.Warpbox.writeClipboard(window.Warpbox.absoluteURL(file.downloadURL));
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
if (action === "download") {
|
if (action === "download") {
|
||||||
|
|||||||
@@ -13,6 +13,7 @@
|
|||||||
const copyURL = document.querySelector("#copy-url");
|
const copyURL = document.querySelector("#copy-url");
|
||||||
const openBox = document.querySelector("#open-box");
|
const openBox = document.querySelector("#open-box");
|
||||||
const manageLink = document.querySelector("#manage-link");
|
const manageLink = document.querySelector("#manage-link");
|
||||||
|
const newUpload = document.querySelector("#new-upload");
|
||||||
const RESUMABLE_SESSIONS_KEY = "warpbox-resumable-sessions";
|
const RESUMABLE_SESSIONS_KEY = "warpbox-resumable-sessions";
|
||||||
|
|
||||||
if (!form || !dropZone || !fileInput) {
|
if (!form || !dropZone || !fileInput) {
|
||||||
@@ -44,6 +45,8 @@
|
|||||||
let latestBoxURL = "";
|
let latestBoxURL = "";
|
||||||
let selectedFiles = [];
|
let selectedFiles = [];
|
||||||
let uploadLocked = false;
|
let uploadLocked = false;
|
||||||
|
let recoveredDraft = null;
|
||||||
|
let resumeMode = false;
|
||||||
|
|
||||||
["dragenter", "dragover"].forEach((eventName) => {
|
["dragenter", "dragover"].forEach((eventName) => {
|
||||||
dropZone.addEventListener(eventName, (event) => {
|
dropZone.addEventListener(eventName, (event) => {
|
||||||
@@ -95,7 +98,11 @@
|
|||||||
|
|
||||||
const submit = form.querySelector("button[type='submit']");
|
const submit = form.querySelector("button[type='submit']");
|
||||||
const formData = uploadFormData();
|
const formData = uploadFormData();
|
||||||
|
if (resumeMode && recoveredDraft) {
|
||||||
|
renderResumeQueue(recoveredDraft.session, selectedFiles);
|
||||||
|
} else {
|
||||||
renderQueue(selectedFiles, "queued");
|
renderQueue(selectedFiles, "queued");
|
||||||
|
}
|
||||||
setLoading(true, submit);
|
setLoading(true, submit);
|
||||||
|
|
||||||
try {
|
try {
|
||||||
@@ -103,8 +110,19 @@
|
|||||||
renderResult(payload);
|
renderResult(payload);
|
||||||
form.reset();
|
form.reset();
|
||||||
selectedFiles = [];
|
selectedFiles = [];
|
||||||
|
resumeMode = false;
|
||||||
|
recoveredDraft = null;
|
||||||
fileInput.value = "";
|
fileInput.value = "";
|
||||||
updateSelectedState();
|
if (uploadQueue) {
|
||||||
|
uploadQueue.hidden = true;
|
||||||
|
uploadQueue.replaceChildren();
|
||||||
|
}
|
||||||
|
if (newUpload) {
|
||||||
|
newUpload.hidden = true;
|
||||||
|
}
|
||||||
|
if (fileSummary) {
|
||||||
|
fileSummary.textContent = "Upload complete.";
|
||||||
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
updateStatus(error.message || "Upload failed");
|
updateStatus(error.message || "Upload failed");
|
||||||
} finally {
|
} finally {
|
||||||
@@ -118,6 +136,16 @@
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (newUpload) {
|
||||||
|
newUpload.addEventListener("click", () => {
|
||||||
|
cancelRecoveredDraft().catch((error) => {
|
||||||
|
updateStatus(error.message || "Upload draft could not be deleted");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
recoverResumableSessions();
|
||||||
|
|
||||||
function addSelectedFiles(files) {
|
function addSelectedFiles(files) {
|
||||||
if (uploadLocked) {
|
if (uploadLocked) {
|
||||||
return;
|
return;
|
||||||
@@ -145,14 +173,25 @@
|
|||||||
title.textContent = count === 0 ? "Drop files to upload" : count === 1 ? "1 file selected" : `${count} files selected`;
|
title.textContent = count === 0 ? "Drop files to upload" : count === 1 ? "1 file selected" : `${count} files selected`;
|
||||||
}
|
}
|
||||||
if (fileSummary) {
|
if (fileSummary) {
|
||||||
|
if (resumeMode && recoveredDraft) {
|
||||||
|
fileSummary.textContent = count === 0
|
||||||
|
? "Reselect missing files to resume, or add extra files to this upload."
|
||||||
|
: `${count} local file${count === 1 ? "" : "s"} ready for the recovered upload.`;
|
||||||
|
} else {
|
||||||
fileSummary.textContent = count === 0 ? "Choose one or more files to begin." : `${count} file${count === 1 ? "" : "s"} ready.`;
|
fileSummary.textContent = count === 0 ? "Choose one or more files to begin." : `${count} file${count === 1 ? "" : "s"} ready.`;
|
||||||
}
|
}
|
||||||
if (count > 0) {
|
}
|
||||||
|
if (resumeMode && recoveredDraft) {
|
||||||
|
renderResumeQueue(recoveredDraft.session, selectedFiles);
|
||||||
|
} else if (count > 0) {
|
||||||
renderQueue(selectedFiles, "queued");
|
renderQueue(selectedFiles, "queued");
|
||||||
} else if (uploadQueue) {
|
} else if (uploadQueue) {
|
||||||
uploadQueue.hidden = true;
|
uploadQueue.hidden = true;
|
||||||
uploadQueue.replaceChildren();
|
uploadQueue.replaceChildren();
|
||||||
}
|
}
|
||||||
|
if (newUpload) {
|
||||||
|
newUpload.hidden = !(resumeMode && recoveredDraft);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function setLoading(isLoading, submit) {
|
function setLoading(isLoading, submit) {
|
||||||
@@ -164,6 +203,9 @@
|
|||||||
submit.disabled = isLoading;
|
submit.disabled = isLoading;
|
||||||
submit.textContent = isLoading ? "Uploading..." : "Upload files";
|
submit.textContent = isLoading ? "Uploading..." : "Upload files";
|
||||||
}
|
}
|
||||||
|
if (newUpload) {
|
||||||
|
newUpload.disabled = isLoading;
|
||||||
|
}
|
||||||
updateStatus(isLoading ? "Transferring files..." : "");
|
updateStatus(isLoading ? "Transferring files..." : "");
|
||||||
setTotalProgress(isLoading ? 0 : 100);
|
setTotalProgress(isLoading ? 0 : 100);
|
||||||
}
|
}
|
||||||
@@ -179,15 +221,15 @@
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
latestBoxURL = payload.boxUrl;
|
latestBoxURL = window.Warpbox.absoluteURL(payload.boxUrl);
|
||||||
result.hidden = false;
|
result.hidden = false;
|
||||||
openBox.href = payload.boxUrl;
|
openBox.href = latestBoxURL;
|
||||||
resultMeta.textContent = `${payload.files.length} file${payload.files.length === 1 ? "" : "s"} · expires ${window.Warpbox.formatDate(payload.expiresAt)}`;
|
resultMeta.textContent = `${payload.files.length} file${payload.files.length === 1 ? "" : "s"} · expires ${window.Warpbox.formatDate(payload.expiresAt)}`;
|
||||||
if (manageLink) {
|
if (manageLink) {
|
||||||
const anchor = manageLink.querySelector("a");
|
const anchor = manageLink.querySelector("a");
|
||||||
manageLink.hidden = !payload.manageUrl;
|
manageLink.hidden = !payload.manageUrl;
|
||||||
if (anchor && payload.manageUrl) {
|
if (anchor && payload.manageUrl) {
|
||||||
anchor.href = payload.manageUrl;
|
anchor.href = window.Warpbox.absoluteURL(payload.manageUrl);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -195,11 +237,12 @@
|
|||||||
payload.files.forEach((file) => {
|
payload.files.forEach((file) => {
|
||||||
resultList.append(createFileRow({
|
resultList.append(createFileRow({
|
||||||
name: file.name,
|
name: file.name,
|
||||||
meta: `${file.size} · ${file.url}`,
|
meta: `${file.size} · ${window.Warpbox.absoluteURL(file.url)}`,
|
||||||
progress: 100,
|
progress: 100,
|
||||||
status: "complete",
|
status: "complete",
|
||||||
}));
|
}));
|
||||||
});
|
});
|
||||||
|
result.scrollIntoView({ behavior: "smooth", block: "start" });
|
||||||
}
|
}
|
||||||
|
|
||||||
function uploadWithProgress(url, formData, files) {
|
function uploadWithProgress(url, formData, files) {
|
||||||
@@ -263,9 +306,22 @@
|
|||||||
collectionId: formData.get("collection_id") || "",
|
collectionId: formData.get("collection_id") || "",
|
||||||
};
|
};
|
||||||
const persistable = !createPayload.password;
|
const persistable = !createPayload.password;
|
||||||
let session = persistable ? await findResumableSession(createPayload) : null;
|
let session = null;
|
||||||
|
if (persistable && resumeMode && recoveredDraft) {
|
||||||
|
session = await fetchResumableStatus(recoveredDraft.session.sessionId, recoveredDraft.session.resumeToken);
|
||||||
|
session.resumeToken = recoveredDraft.session.resumeToken;
|
||||||
|
} else if (persistable) {
|
||||||
|
session = await findResumableSession(createPayload);
|
||||||
|
}
|
||||||
if (session) {
|
if (session) {
|
||||||
|
validateResumeSelection(session, createPayload);
|
||||||
session = await addMissingResumableFiles(session, createPayload);
|
session = await addMissingResumableFiles(session, createPayload);
|
||||||
|
if (resumeMode && recoveredDraft && recoveredDraft.session.sessionId === session.sessionId) {
|
||||||
|
recoveredDraft.session = session;
|
||||||
|
}
|
||||||
|
if (persistable) {
|
||||||
|
saveResumableSession(session, createPayload);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if (!session || session.status !== "uploading") {
|
if (!session || session.status !== "uploading") {
|
||||||
try {
|
try {
|
||||||
@@ -304,7 +360,7 @@
|
|||||||
}
|
}
|
||||||
const start = chunkIndex * session.chunkSize;
|
const start = chunkIndex * session.chunkSize;
|
||||||
const end = Math.min(file.size, start + session.chunkSize);
|
const end = Math.min(file.size, start + session.chunkSize);
|
||||||
await uploadChunk(session.sessionId, sessionFile.id, chunkIndex, file.slice(start, end), (loaded) => {
|
await uploadChunkWithRetry(session, sessionFile, chunkIndex, file.slice(start, end), (loaded) => {
|
||||||
const currentTotal = completedByFile.reduce((sum, bytes) => sum + bytes, 0) + loaded;
|
const currentTotal = completedByFile.reduce((sum, bytes) => sum + bytes, 0) + loaded;
|
||||||
setTotalProgress(percentForBytes(currentTotal, totalBytes));
|
setTotalProgress(percentForBytes(currentTotal, totalBytes));
|
||||||
setSingleFileProgress(fileIndex, file, percentForBytes(completedByFile[fileIndex] + loaded, file.size));
|
setSingleFileProgress(fileIndex, file, percentForBytes(completedByFile[fileIndex] + loaded, file.size));
|
||||||
@@ -320,12 +376,20 @@
|
|||||||
setSingleFileProgress(fileIndex, file, 100);
|
setSingleFileProgress(fileIndex, file, 100);
|
||||||
}
|
}
|
||||||
|
|
||||||
const resultPayload = await completeResumableSession(session.sessionId);
|
updateStatus("Finalizing upload...");
|
||||||
|
const resultPayload = await completeResumableSession(session.sessionId, session.resumeToken);
|
||||||
|
const wasResumeMode = resumeMode;
|
||||||
if (persistable) {
|
if (persistable) {
|
||||||
removeResumableSession(session.sessionId);
|
removeResumableSession(session.sessionId);
|
||||||
}
|
}
|
||||||
|
if (resumeMode && recoveredDraft && recoveredDraft.session.sessionId === session.sessionId) {
|
||||||
|
resumeMode = false;
|
||||||
|
recoveredDraft = null;
|
||||||
|
}
|
||||||
setTotalProgress(100);
|
setTotalProgress(100);
|
||||||
|
if (!wasResumeMode) {
|
||||||
setFileProgress(files, 100);
|
setFileProgress(files, 100);
|
||||||
|
}
|
||||||
return resultPayload;
|
return resultPayload;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -341,17 +405,18 @@
|
|||||||
return readUploadJSON(response, "Upload session could not be created");
|
return readUploadJSON(response, "Upload session could not be created");
|
||||||
}
|
}
|
||||||
|
|
||||||
async function fetchResumableStatus(sessionID) {
|
async function fetchResumableStatus(sessionID, resumeToken) {
|
||||||
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}`, {
|
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}`, {
|
||||||
headers: { "Accept": "application/json" },
|
headers: resumableHeaders(resumeToken),
|
||||||
});
|
});
|
||||||
return readUploadJSON(response, "Upload session could not be resumed");
|
return readUploadJSON(response, "Upload session could not be resumed");
|
||||||
}
|
}
|
||||||
|
|
||||||
async function addResumableFiles(sessionID, files) {
|
async function addResumableFiles(sessionID, resumeToken, files) {
|
||||||
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}/files`, {
|
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}/files`, {
|
||||||
method: "POST",
|
method: "POST",
|
||||||
headers: {
|
headers: {
|
||||||
|
...resumableHeaders(resumeToken),
|
||||||
"Accept": "application/json",
|
"Accept": "application/json",
|
||||||
"Content-Type": "application/json",
|
"Content-Type": "application/json",
|
||||||
},
|
},
|
||||||
@@ -360,11 +425,12 @@
|
|||||||
return readUploadJSON(response, "Upload session files could not be added");
|
return readUploadJSON(response, "Upload session files could not be added");
|
||||||
}
|
}
|
||||||
|
|
||||||
function uploadChunk(sessionID, fileID, chunkIndex, chunk, onProgress) {
|
function uploadChunk(sessionID, resumeToken, fileID, chunkIndex, chunk, onProgress) {
|
||||||
return new Promise((resolve, reject) => {
|
return new Promise((resolve, reject) => {
|
||||||
const request = new XMLHttpRequest();
|
const request = new XMLHttpRequest();
|
||||||
request.open("PUT", `/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}/files/${encodeURIComponent(fileID)}/chunks/${chunkIndex}`);
|
request.open("PUT", `/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}/files/${encodeURIComponent(fileID)}/chunks/${chunkIndex}`);
|
||||||
request.setRequestHeader("Accept", "application/json");
|
request.setRequestHeader("Accept", "application/json");
|
||||||
|
request.setRequestHeader("X-Warpbox-Resume-Token", resumeToken || "");
|
||||||
request.upload.addEventListener("progress", (event) => {
|
request.upload.addEventListener("progress", (event) => {
|
||||||
if (event.lengthComputable && onProgress) {
|
if (event.lengthComputable && onProgress) {
|
||||||
onProgress(event.loaded);
|
onProgress(event.loaded);
|
||||||
@@ -389,14 +455,54 @@
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
async function completeResumableSession(sessionID) {
|
async function uploadChunkWithRetry(session, sessionFile, chunkIndex, chunk, onProgress) {
|
||||||
|
const delays = [1000, 2000, 5000, 10000, 20000];
|
||||||
|
let lastError = null;
|
||||||
|
for (let attempt = 0; attempt <= delays.length; attempt++) {
|
||||||
|
try {
|
||||||
|
return await uploadChunk(session.sessionId, session.resumeToken, sessionFile.id, chunkIndex, chunk, onProgress);
|
||||||
|
} catch (error) {
|
||||||
|
lastError = error;
|
||||||
|
if (attempt >= delays.length) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
const seconds = Math.round(delays[attempt] / 1000);
|
||||||
|
updateStatus(`Connection interrupted, retrying chunk ${chunkIndex + 1} in ${seconds}s`);
|
||||||
|
await wait(delays[attempt]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
throw lastError || new Error("Chunk upload failed");
|
||||||
|
}
|
||||||
|
|
||||||
|
async function completeResumableSession(sessionID, resumeToken) {
|
||||||
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}/complete`, {
|
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}/complete`, {
|
||||||
method: "POST",
|
method: "POST",
|
||||||
headers: { "Accept": "application/json" },
|
headers: resumableHeaders(resumeToken),
|
||||||
});
|
});
|
||||||
return readUploadJSON(response, "Upload could not be completed");
|
return readUploadJSON(response, "Upload could not be completed");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function cancelResumableSession(sessionID, resumeToken) {
|
||||||
|
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}`, {
|
||||||
|
method: "DELETE",
|
||||||
|
headers: resumableHeaders(resumeToken),
|
||||||
|
});
|
||||||
|
if (!response.ok && response.status !== 404) {
|
||||||
|
await readUploadJSON(response, "Upload draft could not be deleted");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function resumableHeaders(resumeToken) {
|
||||||
|
return {
|
||||||
|
"Accept": "application/json",
|
||||||
|
"X-Warpbox-Resume-Token": resumeToken || "",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function wait(ms) {
|
||||||
|
return new Promise((resolve) => window.setTimeout(resolve, ms));
|
||||||
|
}
|
||||||
|
|
||||||
async function readUploadJSON(response, fallback) {
|
async function readUploadJSON(response, fallback) {
|
||||||
let payload = {};
|
let payload = {};
|
||||||
try {
|
try {
|
||||||
@@ -421,15 +527,15 @@
|
|||||||
if (!record.files || !record.files.some((file) => selectedKeys.has(resumableFileKey(file)))) {
|
if (!record.files || !record.files.some((file) => selectedKeys.has(resumableFileKey(file)))) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
const session = await fetchResumableStatus(record.sessionId).catch(() => null);
|
const session = await fetchResumableStatus(record.sessionId, record.resumeToken).catch(() => null);
|
||||||
if (!session || session.status !== "uploading") {
|
if (!session || session.status !== "uploading") {
|
||||||
removeResumableSession(record.sessionId);
|
removeResumableSession(record.sessionId);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
session.resumeToken = record.resumeToken;
|
||||||
const sessionKeys = new Set(session.files.map((file) => resumableFileKey(file)));
|
const sessionKeys = new Set(session.files.map((file) => resumableFileKey(file)));
|
||||||
const sessionHasOnlySelectedFiles = session.files.every((file) => selectedKeys.has(resumableFileKey(file)));
|
|
||||||
const selectedContainsSessionFile = Array.from(sessionKeys).some((key) => selectedKeys.has(key));
|
const selectedContainsSessionFile = Array.from(sessionKeys).some((key) => selectedKeys.has(key));
|
||||||
if (sessionHasOnlySelectedFiles && selectedContainsSessionFile) {
|
if (selectedContainsSessionFile) {
|
||||||
return session;
|
return session;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -442,7 +548,25 @@
|
|||||||
if (missing.length === 0) {
|
if (missing.length === 0) {
|
||||||
return session;
|
return session;
|
||||||
}
|
}
|
||||||
return addResumableFiles(session.sessionId, missing);
|
const updated = await addResumableFiles(session.sessionId, session.resumeToken, missing);
|
||||||
|
updated.resumeToken = session.resumeToken;
|
||||||
|
return updated;
|
||||||
|
}
|
||||||
|
|
||||||
|
function validateResumeSelection(session, payload) {
|
||||||
|
if (!resumeMode || !recoveredDraft || session.sessionId !== recoveredDraft.session.sessionId) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const existingByNameSize = new Map();
|
||||||
|
(session.files || []).forEach((file) => {
|
||||||
|
existingByNameSize.set(`${file.name}:${file.size}`, resumableFileKey(file));
|
||||||
|
});
|
||||||
|
for (const file of payload.files || []) {
|
||||||
|
const expectedKey = existingByNameSize.get(`${file.name}:${file.size}`);
|
||||||
|
if (expectedKey && expectedKey !== resumableFileKey(file)) {
|
||||||
|
throw new Error(`"${file.name}" does not match the pending upload. Select the exact original file.`);
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function matchSessionFile(session, file) {
|
function matchSessionFile(session, file) {
|
||||||
@@ -478,11 +602,21 @@
|
|||||||
const records = loadResumableSessions().filter((record) => record.sessionId !== session.sessionId);
|
const records = loadResumableSessions().filter((record) => record.sessionId !== session.sessionId);
|
||||||
records.push({
|
records.push({
|
||||||
sessionId: session.sessionId,
|
sessionId: session.sessionId,
|
||||||
|
resumeToken: session.resumeToken || "",
|
||||||
optionKey: resumableOptionKey(payload),
|
optionKey: resumableOptionKey(payload),
|
||||||
|
options: {
|
||||||
|
expiresMinutes: payload.expiresMinutes,
|
||||||
|
maxDownloads: payload.maxDownloads,
|
||||||
|
obfuscateMetadata: !!payload.obfuscateMetadata,
|
||||||
|
collectionId: payload.collectionId || "",
|
||||||
|
},
|
||||||
files: session.files.map((file) => ({
|
files: session.files.map((file) => ({
|
||||||
name: file.name,
|
name: file.name,
|
||||||
size: file.size,
|
size: file.size,
|
||||||
|
contentType: file.contentType || "application/octet-stream",
|
||||||
fingerprint: file.fingerprint || "",
|
fingerprint: file.fingerprint || "",
|
||||||
|
uploadedChunks: file.uploadedChunks || [],
|
||||||
|
chunkCount: file.chunkCount || 0,
|
||||||
})),
|
})),
|
||||||
updatedAt: new Date().toISOString(),
|
updatedAt: new Date().toISOString(),
|
||||||
});
|
});
|
||||||
@@ -492,6 +626,38 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function recoverResumableSessions() {
|
||||||
|
const records = loadResumableSessions()
|
||||||
|
.filter((record) => record.sessionId && record.resumeToken)
|
||||||
|
.sort((a, b) => new Date(b.updatedAt || 0).getTime() - new Date(a.updatedAt || 0).getTime());
|
||||||
|
if (records.length === 0) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
for (const record of records) {
|
||||||
|
const session = await fetchResumableStatus(record.sessionId, record.resumeToken).catch(() => null);
|
||||||
|
if (!session || session.status !== "uploading") {
|
||||||
|
removeResumableSession(record.sessionId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
session.resumeToken = record.resumeToken;
|
||||||
|
recoveredDraft = { session, record };
|
||||||
|
selectedFiles = [];
|
||||||
|
renderRecoveredQueue([{ session, record }]);
|
||||||
|
updateRecoveredSummary(session);
|
||||||
|
showRecoveryModal(recoveredDraft);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function updateRecoveredSummary(session) {
|
||||||
|
updateStatus("Unfinished upload found. Choose how to continue.");
|
||||||
|
if (fileSummary) {
|
||||||
|
const totalFiles = (session.files || []).length;
|
||||||
|
const completedFiles = completedSessionFiles(session).length;
|
||||||
|
fileSummary.textContent = `Recovered ${totalFiles} pending file${totalFiles === 1 ? "" : "s"}; ${completedFiles} fully uploaded.`;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
function removeResumableSession(sessionID) {
|
function removeResumableSession(sessionID) {
|
||||||
try {
|
try {
|
||||||
const records = loadResumableSessions().filter((record) => record.sessionId !== sessionID);
|
const records = loadResumableSessions().filter((record) => record.sessionId !== sessionID);
|
||||||
@@ -501,6 +667,105 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function completedSessionFiles(session) {
|
||||||
|
return (session.files || []).filter((file) => (file.uploadedChunks || []).length >= file.chunkCount);
|
||||||
|
}
|
||||||
|
|
||||||
|
function showRecoveryModal(draft) {
|
||||||
|
const old = document.querySelector(".upload-recovery-overlay");
|
||||||
|
if (old) {
|
||||||
|
old.remove();
|
||||||
|
}
|
||||||
|
const completeCount = completedSessionFiles(draft.session).length;
|
||||||
|
const totalCount = (draft.session.files || []).length;
|
||||||
|
const overlay = document.createElement("div");
|
||||||
|
overlay.className = "upload-recovery-overlay";
|
||||||
|
overlay.setAttribute("role", "dialog");
|
||||||
|
overlay.setAttribute("aria-modal", "true");
|
||||||
|
overlay.setAttribute("aria-labelledby", "upload-recovery-title");
|
||||||
|
|
||||||
|
const modal = document.createElement("div");
|
||||||
|
modal.className = "upload-recovery-modal card";
|
||||||
|
const content = document.createElement("div");
|
||||||
|
content.className = "card-content";
|
||||||
|
|
||||||
|
const title = document.createElement("h2");
|
||||||
|
title.id = "upload-recovery-title";
|
||||||
|
title.textContent = "Unfinished upload found";
|
||||||
|
const copy = document.createElement("p");
|
||||||
|
copy.textContent = `Warpbox found a private draft with ${totalCount} file${totalCount === 1 ? "" : "s"}. ${completeCount} file${completeCount === 1 ? " is" : "s are"} already fully uploaded.`;
|
||||||
|
|
||||||
|
const actions = document.createElement("div");
|
||||||
|
actions.className = "upload-recovery-actions";
|
||||||
|
|
||||||
|
const startOver = document.createElement("button");
|
||||||
|
startOver.type = "button";
|
||||||
|
startOver.className = "button button-danger";
|
||||||
|
startOver.textContent = "New Upload";
|
||||||
|
startOver.addEventListener("click", async () => {
|
||||||
|
startOver.disabled = true;
|
||||||
|
try {
|
||||||
|
await cancelRecoveredDraft();
|
||||||
|
overlay.remove();
|
||||||
|
} catch (error) {
|
||||||
|
startOver.disabled = false;
|
||||||
|
updateStatus(error.message || "Upload draft could not be deleted");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const resume = document.createElement("button");
|
||||||
|
resume.type = "button";
|
||||||
|
resume.className = "button button-primary";
|
||||||
|
resume.textContent = "Resume";
|
||||||
|
resume.addEventListener("click", () => {
|
||||||
|
resumeRecoveredDraft();
|
||||||
|
overlay.remove();
|
||||||
|
});
|
||||||
|
|
||||||
|
actions.append(startOver, resume);
|
||||||
|
content.append(title, copy, actions);
|
||||||
|
modal.append(content);
|
||||||
|
overlay.append(modal);
|
||||||
|
document.body.append(overlay);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function cancelRecoveredDraft() {
|
||||||
|
if (!recoveredDraft) {
|
||||||
|
resetFreshUploadState();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const draft = recoveredDraft;
|
||||||
|
updateStatus("Deleting unfinished upload...");
|
||||||
|
await cancelResumableSession(draft.session.sessionId, draft.session.resumeToken);
|
||||||
|
removeResumableSession(draft.session.sessionId);
|
||||||
|
resetFreshUploadState();
|
||||||
|
}
|
||||||
|
|
||||||
|
function resumeRecoveredDraft() {
|
||||||
|
if (!recoveredDraft) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
resumeMode = true;
|
||||||
|
selectedFiles = [];
|
||||||
|
renderResumeQueue(recoveredDraft.session, selectedFiles);
|
||||||
|
updateSelectedState();
|
||||||
|
updateStatus("Drop or reselect missing files to continue. Extra files will be added to this upload.");
|
||||||
|
}
|
||||||
|
|
||||||
|
function resetFreshUploadState() {
|
||||||
|
selectedFiles = [];
|
||||||
|
resumeMode = false;
|
||||||
|
recoveredDraft = null;
|
||||||
|
fileInput.value = "";
|
||||||
|
result.hidden = true;
|
||||||
|
if (resultList) {
|
||||||
|
resultList.replaceChildren();
|
||||||
|
}
|
||||||
|
setTotalProgress(0);
|
||||||
|
updateStatus("");
|
||||||
|
updateSelectedState();
|
||||||
|
}
|
||||||
|
|
||||||
function uploadedBytesForSessionFile(file, chunkSize) {
|
function uploadedBytesForSessionFile(file, chunkSize) {
|
||||||
return (file.uploadedChunks || []).reduce((sum, index) => {
|
return (file.uploadedChunks || []).reduce((sum, index) => {
|
||||||
const start = index * chunkSize;
|
const start = index * chunkSize;
|
||||||
@@ -509,6 +774,91 @@
|
|||||||
}, 0);
|
}, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function renderRecoveredQueue(items) {
|
||||||
|
if (!uploadQueue) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const rows = [];
|
||||||
|
items.forEach(({ session }) => {
|
||||||
|
(session.files || []).forEach((file) => {
|
||||||
|
const uploadedBytes = uploadedBytesForSessionFile(file, session.chunkSize);
|
||||||
|
const complete = (file.uploadedChunks || []).length >= file.chunkCount;
|
||||||
|
rows.push({
|
||||||
|
name: file.name,
|
||||||
|
size: file.size,
|
||||||
|
uploadedBytes,
|
||||||
|
meta: complete
|
||||||
|
? `${window.Warpbox.formatBytes(file.size)} · uploaded`
|
||||||
|
: `${window.Warpbox.formatBytes(uploadedBytes)} of ${window.Warpbox.formatBytes(file.size)} · Drop/reselect this file to continue`,
|
||||||
|
progress: percentForBytes(uploadedBytes, file.size),
|
||||||
|
status: complete ? "complete" : "waiting",
|
||||||
|
readonly: true,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
uploadQueue.hidden = rows.length === 0;
|
||||||
|
uploadQueue.replaceChildren();
|
||||||
|
rows.forEach((row) => uploadQueue.append(createFileRow(row)));
|
||||||
|
const totalBytes = rows.reduce((sum, row) => sum + (row.size || 0), 0);
|
||||||
|
if (totalBytes > 0) {
|
||||||
|
setTotalProgress(percentForBytes(rows.reduce((sum, row) => sum + (row.uploadedBytes || 0), 0), totalBytes));
|
||||||
|
} else if (rows.length > 0) {
|
||||||
|
const completed = rows.filter((row) => row.status === "complete").length;
|
||||||
|
setTotalProgress(percentForBytes(completed, rows.length));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderResumeQueue(session, localFiles) {
|
||||||
|
if (!uploadQueue) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const rows = [];
|
||||||
|
const localByNameSize = new Map();
|
||||||
|
(localFiles || []).forEach((file, index) => {
|
||||||
|
localByNameSize.set(`${file.name}:${file.size}`, { file, index });
|
||||||
|
});
|
||||||
|
const usedLocalIndexes = new Set();
|
||||||
|
(session.files || []).forEach((file) => {
|
||||||
|
const uploadedBytes = uploadedBytesForSessionFile(file, session.chunkSize);
|
||||||
|
const complete = (file.uploadedChunks || []).length >= file.chunkCount;
|
||||||
|
const localMatch = localByNameSize.get(`${file.name}:${file.size}`) || null;
|
||||||
|
if (localMatch) {
|
||||||
|
usedLocalIndexes.add(localMatch.index);
|
||||||
|
}
|
||||||
|
rows.push({
|
||||||
|
name: file.name,
|
||||||
|
size: file.size,
|
||||||
|
uploadedBytes,
|
||||||
|
meta: complete
|
||||||
|
? `${window.Warpbox.formatBytes(file.size)} · uploaded`
|
||||||
|
: localMatch
|
||||||
|
? `${window.Warpbox.formatBytes(uploadedBytes)} of ${window.Warpbox.formatBytes(file.size)} · ready to resume`
|
||||||
|
: `${window.Warpbox.formatBytes(uploadedBytes)} of ${window.Warpbox.formatBytes(file.size)} · waiting for local file`,
|
||||||
|
progress: percentForBytes(uploadedBytes, file.size),
|
||||||
|
status: complete ? "complete" : localMatch ? "queued" : "waiting",
|
||||||
|
readonly: !localMatch,
|
||||||
|
index: localMatch ? localMatch.index : undefined,
|
||||||
|
removable: Boolean(localMatch && !complete),
|
||||||
|
});
|
||||||
|
});
|
||||||
|
(localFiles || []).forEach((file, index) => {
|
||||||
|
if (usedLocalIndexes.has(index)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
rows.push({
|
||||||
|
name: file.name,
|
||||||
|
meta: `${window.Warpbox.formatBytes(file.size)} · new file`,
|
||||||
|
progress: 0,
|
||||||
|
status: "queued",
|
||||||
|
index,
|
||||||
|
removable: true,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
uploadQueue.hidden = rows.length === 0;
|
||||||
|
uploadQueue.replaceChildren();
|
||||||
|
rows.forEach((row) => uploadQueue.append(createFileRow(row)));
|
||||||
|
}
|
||||||
|
|
||||||
function percentForBytes(bytes, total) {
|
function percentForBytes(bytes, total) {
|
||||||
if (!total) {
|
if (!total) {
|
||||||
return 100;
|
return 100;
|
||||||
@@ -536,9 +886,11 @@
|
|||||||
|
|
||||||
function createFileRow(file) {
|
function createFileRow(file) {
|
||||||
const row = document.createElement("div");
|
const row = document.createElement("div");
|
||||||
row.className = "result-item upload-file-row";
|
row.className = `result-item upload-file-row upload-file-${file.status || "queued"}`;
|
||||||
row.dataset.fileName = file.name;
|
row.dataset.fileName = file.name;
|
||||||
row.dataset.fileIndex = file.index || 0;
|
if (typeof file.index === "number") {
|
||||||
|
row.dataset.fileIndex = file.index;
|
||||||
|
}
|
||||||
|
|
||||||
const body = document.createElement("span");
|
const body = document.createElement("span");
|
||||||
const name = document.createElement("strong");
|
const name = document.createElement("strong");
|
||||||
@@ -560,6 +912,12 @@
|
|||||||
fill.style.transform = `scaleX(${file.progress / 100})`;
|
fill.style.transform = `scaleX(${file.progress / 100})`;
|
||||||
bar.append(fill);
|
bar.append(fill);
|
||||||
side.append(percent, bar);
|
side.append(percent, bar);
|
||||||
|
if (file.status === "waiting") {
|
||||||
|
const badge = document.createElement("small");
|
||||||
|
badge.className = "upload-file-state";
|
||||||
|
badge.textContent = "Needs local file";
|
||||||
|
side.append(badge);
|
||||||
|
}
|
||||||
if (file.removable) {
|
if (file.removable) {
|
||||||
const remove = document.createElement("button");
|
const remove = document.createElement("button");
|
||||||
remove.className = "upload-file-remove";
|
remove.className = "upload-file-remove";
|
||||||
|
|||||||
@@ -126,6 +126,33 @@
|
|||||||
</label>
|
</label>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<div class="settings-section">
|
||||||
|
<h3 class="settings-section-title">Resumable uploads</h3>
|
||||||
|
<label class="checkbox-field">
|
||||||
|
<input type="checkbox" name="resumable_uploads_enabled" {{if .Data.Settings.ResumableUploadsEnabled}}checked{{end}}>
|
||||||
|
<span>Enable browser/API resumable uploads</span>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>Chunk size (MB)</span>
|
||||||
|
<input name="resumable_chunk_size_mb" value="{{.Data.Settings.ResumableChunkSizeMB}}" required>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>Draft retention (hours)</span>
|
||||||
|
<input type="number" name="resumable_retention_hours" min="1" value="{{.Data.Settings.ResumableRetentionHours}}" required>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>Chunk storage</span>
|
||||||
|
<select name="resumable_chunk_mode" required>
|
||||||
|
<option value="same" {{if eq .Data.Settings.ResumableChunkMode "same"}}selected{{end}}>Default local data path</option>
|
||||||
|
<option value="custom" {{if eq .Data.Settings.ResumableChunkMode "custom"}}selected{{end}}>Custom local path, e.g. fast SSD</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
<span>Custom chunk path</span>
|
||||||
|
<input name="resumable_chunk_path" value="{{.Data.Settings.ResumableChunkPath}}" placeholder="/mnt/fast-ssd/warpbox-chunks">
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
|
|
||||||
<button class="button button-primary" type="submit">Save settings</button>
|
<button class="button button-primary" type="submit">Save settings</button>
|
||||||
</form>
|
</form>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -24,6 +24,12 @@
|
|||||||
{{end}}
|
{{end}}
|
||||||
|
|
||||||
{{if .Data.Files}}
|
{{if .Data.Files}}
|
||||||
|
{{$processing := false}}{{range .Data.Files}}{{if .Processing}}{{$processing = true}}{{end}}{{end}}
|
||||||
|
{{if $processing}}
|
||||||
|
<div class="upload-processing-alert" role="status">
|
||||||
|
Some files are still processing. You can share this link now, but processing files will become available shortly.
|
||||||
|
</div>
|
||||||
|
{{end}}
|
||||||
{{$single := eq (len .Data.Files) 1}}
|
{{$single := eq (len .Data.Files) 1}}
|
||||||
<div class="badge-row">
|
<div class="badge-row">
|
||||||
<span class="badge badge-expiry">Expires {{.Data.ExpiresLabel}}</span>
|
<span class="badge badge-expiry">Expires {{.Data.ExpiresLabel}}</span>
|
||||||
@@ -74,8 +80,8 @@
|
|||||||
</div>
|
</div>
|
||||||
<div class="download-list file-browser is-thumbs" data-file-browser>
|
<div class="download-list file-browser is-thumbs" data-file-browser>
|
||||||
{{range .Data.Files}}
|
{{range .Data.Files}}
|
||||||
<article class="download-item file-card" data-kind="{{.PreviewKind}}" data-file-context data-preview-url="{{.URL}}" data-view-url="{{.DownloadURL}}?inline=1" data-download-url="{{.DownloadURL}}" data-file-name="{{.Name}}" data-reaction-card data-react-url="{{.ReactURL}}" data-reacted="{{if .Reacted}}true{{else}}false{{end}}">
|
<article class="download-item file-card {{if .Processing}}is-processing{{end}}" data-kind="{{.PreviewKind}}" {{if not .Processing}}data-file-context data-preview-url="{{.URL}}" data-view-url="{{.DownloadURL}}?inline=1" data-download-url="{{.DownloadURL}}"{{end}} data-file-name="{{.Name}}" data-reaction-card data-react-url="{{.ReactURL}}" data-reacted="{{if .Reacted}}true{{else}}false{{end}}">
|
||||||
<a class="file-open" href="{{.DownloadURL}}?inline=1"{{if not $single}} target="_blank" rel="noopener noreferrer"{{end}} aria-label="Open {{.Name}}">
|
{{if .Processing}}<div class="file-open" aria-label="{{.Name}} is processing">{{else}}<a class="file-open" href="{{.DownloadURL}}?inline=1"{{if not $single}} target="_blank" rel="noopener noreferrer"{{end}} aria-label="Open {{.Name}}">{{end}}
|
||||||
<span class="file-media">
|
<span class="file-media">
|
||||||
{{if .HasThumbnail}}
|
{{if .HasThumbnail}}
|
||||||
<img class="file-thumb" src="{{.ThumbnailURL}}" alt="" loading="lazy">
|
<img class="file-thumb" src="{{.ThumbnailURL}}" alt="" loading="lazy">
|
||||||
@@ -86,11 +92,11 @@
|
|||||||
</span>
|
</span>
|
||||||
<span class="file-main">
|
<span class="file-main">
|
||||||
<strong class="file-name" title="{{.Name}}">{{.Name}}</strong>
|
<strong class="file-name" title="{{.Name}}">{{.Name}}</strong>
|
||||||
<small>{{.Size}} · {{.ContentType}}</small>
|
<small>{{.Size}} · {{if .Processing}}Processing{{else}}{{.ContentType}}{{end}}</small>
|
||||||
</span>
|
</span>
|
||||||
<span class="file-type">{{.ContentType}}</span>
|
<span class="file-type">{{if .Processing}}Processing{{else}}{{.ContentType}}{{end}}</span>
|
||||||
<span class="file-size">{{.Size}}</span>
|
<span class="file-size">{{.Size}}</span>
|
||||||
</a>
|
{{if .Processing}}</div>{{else}}</a>{{end}}
|
||||||
{{if not $.Data.Locked}}
|
{{if not $.Data.Locked}}
|
||||||
<div class="file-reaction-dock" data-reaction-dock>
|
<div class="file-reaction-dock" data-reaction-dock>
|
||||||
<div class="file-reactions" data-reaction-list>
|
<div class="file-reactions" data-reaction-list>
|
||||||
|
|||||||
@@ -77,6 +77,7 @@
|
|||||||
<div class="form-footer">
|
<div class="form-footer">
|
||||||
<p id="file-summary">Choose one or more files to begin.</p>
|
<p id="file-summary">Choose one or more files to begin.</p>
|
||||||
<button class="button button-primary" type="submit">Upload files</button>
|
<button class="button button-primary" type="submit">Upload files</button>
|
||||||
|
<button class="button button-danger upload-new-button" type="button" id="new-upload" hidden>New upload</button>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
2
scripts/env/dev.env.example
vendored
2
scripts/env/dev.env.example
vendored
@@ -12,6 +12,8 @@ WARPBOX_THUMBNAIL_EVERY=1m
|
|||||||
WARPBOX_RESUMABLE_UPLOADS_ENABLED=true
|
WARPBOX_RESUMABLE_UPLOADS_ENABLED=true
|
||||||
WARPBOX_RESUMABLE_CHUNK_MB=16
|
WARPBOX_RESUMABLE_CHUNK_MB=16
|
||||||
WARPBOX_RESUMABLE_RETENTION_HOURS=1
|
WARPBOX_RESUMABLE_RETENTION_HOURS=1
|
||||||
|
WARPBOX_RESUMABLE_CHUNK_MODE=same
|
||||||
|
WARPBOX_RESUMABLE_CHUNK_PATH=
|
||||||
WARPBOX_MAX_UPLOAD_SIZE_MB=16384
|
WARPBOX_MAX_UPLOAD_SIZE_MB=16384
|
||||||
WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true
|
WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true
|
||||||
WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512
|
WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512
|
||||||
|
|||||||
Reference in New Issue
Block a user