feat(uploads): add native resumable upload support
Implement a native chunked resumable upload API and frontend integration to support reliable large file uploads. Changes include: - Added a 3-step resumable upload API flow (create session, upload chunks, complete session). - Introduced configuration options for chunk size, retention hours, and toggling the feature. - Updated the frontend to utilize resumable uploads with progress tracking. - Configured temporary chunk storage under `data/tmp/uploads` with automatic cleanup. - Documented the API flow and configuration in the README.
This commit is contained in:
@@ -9,6 +9,9 @@ WARPBOX_CLEANUP_ENABLED=true
|
|||||||
WARPBOX_CLEANUP_EVERY=1h
|
WARPBOX_CLEANUP_EVERY=1h
|
||||||
WARPBOX_THUMBNAIL_ENABLED=true
|
WARPBOX_THUMBNAIL_ENABLED=true
|
||||||
WARPBOX_THUMBNAIL_EVERY=1m
|
WARPBOX_THUMBNAIL_EVERY=1m
|
||||||
|
WARPBOX_RESUMABLE_UPLOADS_ENABLED=true
|
||||||
|
WARPBOX_RESUMABLE_CHUNK_MB=64
|
||||||
|
WARPBOX_RESUMABLE_RETENTION_HOURS=1
|
||||||
WARPBOX_MAX_UPLOAD_SIZE_MB=16384
|
WARPBOX_MAX_UPLOAD_SIZE_MB=16384
|
||||||
WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true
|
WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true
|
||||||
WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512
|
WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512
|
||||||
|
|||||||
33
README.md
33
README.md
@@ -33,6 +33,9 @@ Upload policy defaults are also configured in megabytes and can later be changed
|
|||||||
- `WARPBOX_SHORT_WINDOW_SECONDS=60`
|
- `WARPBOX_SHORT_WINDOW_SECONDS=60`
|
||||||
- `WARPBOX_ANONYMOUS_STORAGE_BACKEND=local`
|
- `WARPBOX_ANONYMOUS_STORAGE_BACKEND=local`
|
||||||
- `WARPBOX_USER_STORAGE_BACKEND=local`
|
- `WARPBOX_USER_STORAGE_BACKEND=local`
|
||||||
|
- `WARPBOX_RESUMABLE_UPLOADS_ENABLED=true`
|
||||||
|
- `WARPBOX_RESUMABLE_CHUNK_MB=8`
|
||||||
|
- `WARPBOX_RESUMABLE_RETENTION_HOURS=24`
|
||||||
- `WARPBOX_TRUSTED_PROXIES=` controls whether forwarded client IP headers are accepted only from specific proxy IPs/CIDRs. See [SECURITY_PROXY.md](./SECURITY_PROXY.md).
|
- `WARPBOX_TRUSTED_PROXIES=` controls whether forwarded client IP headers are accepted only from specific proxy IPs/CIDRs. See [SECURITY_PROXY.md](./SECURITY_PROXY.md).
|
||||||
|
|
||||||
Runtime data is configured with `WARPBOX_DATA_DIR` and defaults to `./data` in the dev environment.
|
Runtime data is configured with `WARPBOX_DATA_DIR` and defaults to `./data` in the dev environment.
|
||||||
@@ -43,6 +46,10 @@ Large uploads are expected to take minutes on normal home/server connections. Ke
|
|||||||
mid-upload; `WARPBOX_READ_HEADER_TIMEOUT=15s` still protects header reads from slowloris-style
|
mid-upload; `WARPBOX_READ_HEADER_TIMEOUT=15s` still protects header reads from slowloris-style
|
||||||
connections.
|
connections.
|
||||||
|
|
||||||
|
Browser uploads use Warpbox-native resumable uploads by default. Chunks are stored temporarily under
|
||||||
|
`data/tmp/uploads/{session_id}` and then streamed into the selected storage backend when the upload
|
||||||
|
is completed. Stale sessions are cleaned by the cleanup job after `WARPBOX_RESUMABLE_RETENTION_HOURS`.
|
||||||
|
|
||||||
Background jobs are enabled with `WARPBOX_JOBS_ENABLED=true`. Individual jobs can be toggled with
|
Background jobs are enabled with `WARPBOX_JOBS_ENABLED=true`. Individual jobs can be toggled with
|
||||||
`WARPBOX_CLEANUP_ENABLED` and `WARPBOX_THUMBNAIL_ENABLED`, and their schedules are configured with
|
`WARPBOX_CLEANUP_ENABLED` and `WARPBOX_THUMBNAIL_ENABLED`, and their schedules are configured with
|
||||||
`WARPBOX_CLEANUP_EVERY` and `WARPBOX_THUMBNAIL_EVERY`.
|
`WARPBOX_CLEANUP_EVERY` and `WARPBOX_THUMBNAIL_EVERY`.
|
||||||
@@ -242,6 +249,31 @@ ShareX selection — which ShareX sends as separate back-to-back requests — la
|
|||||||
link. The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file.
|
link. The shipped `.sxcu` sets `X-Warpbox-Batch: sharex`; remove that header for one box per file.
|
||||||
Requests without the header behave exactly as before.
|
Requests without the header behave exactly as before.
|
||||||
|
|
||||||
|
### Resumable API flow
|
||||||
|
|
||||||
|
Custom clients can use the resumable JSON API for large files:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Create a resumable session from file metadata.
|
||||||
|
curl -s http://localhost:8080/api/v1/uploads/resumable \
|
||||||
|
-H 'Accept: application/json' \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{"files":[{"name":"large.bin","size":1048576,"contentType":"application/octet-stream"}],"expiresMinutes":1440}'
|
||||||
|
|
||||||
|
# 2. Upload exact-sized chunks using the returned sessionId, file id, and chunkSize.
|
||||||
|
dd if=./large.bin bs=8388608 count=1 skip=0 2>/dev/null | \
|
||||||
|
curl -X PUT --data-binary @- \
|
||||||
|
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
|
||||||
|
|
||||||
|
# 3. Complete the session after all chunks are present.
|
||||||
|
curl -X POST -H 'Accept: application/json' \
|
||||||
|
http://localhost:8080/api/v1/uploads/resumable/SESSION_ID/complete
|
||||||
|
```
|
||||||
|
|
||||||
|
The complete response is the same JSON shape as `POST /api/v1/upload`, including `boxUrl`,
|
||||||
|
`manageUrl`, `deleteUrl`, and file URLs. Send `Authorization: Bearer <token>` on every resumable
|
||||||
|
request to upload as an account.
|
||||||
|
|
||||||
## Stage 4 Accounts + Personal Boxes
|
## Stage 4 Accounts + Personal Boxes
|
||||||
|
|
||||||
- `/register` bootstraps the first admin account only when no users exist.
|
- `/register` bootstraps the first admin account only when no users exist.
|
||||||
@@ -275,6 +307,7 @@ Warpbox keeps local runtime data under the configured data directory:
|
|||||||
|
|
||||||
- `data/files/{box_id}/@each@{file_id}.ext` - uploaded file contents when the local backend is selected.
|
- `data/files/{box_id}/@each@{file_id}.ext` - uploaded file contents when the local backend is selected.
|
||||||
- `data/files/{box_id}/@thumb@{file_id}.jpg` - generated previews when the local backend is selected.
|
- `data/files/{box_id}/@thumb@{file_id}.jpg` - generated previews when the local backend is selected.
|
||||||
|
- `data/tmp/uploads/{session_id}` - temporary chunks for unfinished resumable uploads.
|
||||||
- `data/db/warpbox.bbolt` - bbolt metadata database for boxes and file records.
|
- `data/db/warpbox.bbolt` - bbolt metadata database for boxes and file records.
|
||||||
- `data/db/warpbox.bbolt` also stores users, sessions, invites, and collections.
|
- `data/db/warpbox.bbolt` also stores users, sessions, invites, and collections.
|
||||||
- `data/db/warpbox.bbolt` stores upload policy settings and daily usage records keyed by plain IP
|
- `data/db/warpbox.bbolt` stores upload policy settings and daily usage records keyed by plain IP
|
||||||
|
|||||||
@@ -11,27 +11,30 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type Config struct {
|
type Config struct {
|
||||||
AppName string
|
AppName string
|
||||||
AppVersion string
|
AppVersion string
|
||||||
Environment string
|
Environment string
|
||||||
Addr string
|
Addr string
|
||||||
BaseURL string
|
BaseURL string
|
||||||
DataDir string
|
DataDir string
|
||||||
AdminToken string
|
AdminToken string
|
||||||
StaticDir string
|
StaticDir string
|
||||||
TemplateDir string
|
TemplateDir string
|
||||||
ReadHeaderTimeout time.Duration
|
ReadHeaderTimeout time.Duration
|
||||||
ReadTimeout time.Duration
|
ReadTimeout time.Duration
|
||||||
WriteTimeout time.Duration
|
WriteTimeout time.Duration
|
||||||
IdleTimeout time.Duration
|
IdleTimeout time.Duration
|
||||||
TrustedProxies []string
|
TrustedProxies []string
|
||||||
JobsEnabled bool
|
JobsEnabled bool
|
||||||
CleanupEnabled bool
|
CleanupEnabled bool
|
||||||
CleanupEvery time.Duration
|
CleanupEvery time.Duration
|
||||||
ThumbnailEnabled bool
|
ThumbnailEnabled bool
|
||||||
ThumbnailEvery time.Duration
|
ThumbnailEvery time.Duration
|
||||||
MaxUploadSize int64
|
ResumableUploadsEnabled bool
|
||||||
DefaultSettings SettingsDefaults
|
ResumableChunkSize int64
|
||||||
|
ResumableRetention time.Duration
|
||||||
|
MaxUploadSize int64
|
||||||
|
DefaultSettings SettingsDefaults
|
||||||
}
|
}
|
||||||
|
|
||||||
type SettingsDefaults struct {
|
type SettingsDefaults struct {
|
||||||
@@ -56,26 +59,29 @@ type SettingsDefaults struct {
|
|||||||
|
|
||||||
func Load() (Config, error) {
|
func Load() (Config, error) {
|
||||||
cfg := Config{
|
cfg := Config{
|
||||||
AppName: envString("WARPBOX_APP_NAME", "warpbox.dev"),
|
AppName: envString("WARPBOX_APP_NAME", "warpbox.dev"),
|
||||||
AppVersion: envString("APP_VERSION", "dev"),
|
AppVersion: envString("APP_VERSION", "dev"),
|
||||||
Environment: envString("WARPBOX_ENV", "development"),
|
Environment: envString("WARPBOX_ENV", "development"),
|
||||||
Addr: envString("WARPBOX_ADDR", ":8080"),
|
Addr: envString("WARPBOX_ADDR", ":8080"),
|
||||||
BaseURL: strings.TrimRight(envString("WARPBOX_BASE_URL", "http://localhost:8080"), "/"),
|
BaseURL: strings.TrimRight(envString("WARPBOX_BASE_URL", "http://localhost:8080"), "/"),
|
||||||
DataDir: envString("WARPBOX_DATA_DIR", defaultPath("data")),
|
DataDir: envString("WARPBOX_DATA_DIR", defaultPath("data")),
|
||||||
AdminToken: envString("WARPBOX_ADMIN_TOKEN", ""),
|
AdminToken: envString("WARPBOX_ADMIN_TOKEN", ""),
|
||||||
StaticDir: envString("WARPBOX_STATIC_DIR", defaultPath("static")),
|
StaticDir: envString("WARPBOX_STATIC_DIR", defaultPath("static")),
|
||||||
TemplateDir: envString("WARPBOX_TEMPLATE_DIR", defaultPath("templates")),
|
TemplateDir: envString("WARPBOX_TEMPLATE_DIR", defaultPath("templates")),
|
||||||
ReadHeaderTimeout: envDuration("WARPBOX_READ_HEADER_TIMEOUT", 15*time.Second),
|
ReadHeaderTimeout: envDuration("WARPBOX_READ_HEADER_TIMEOUT", 15*time.Second),
|
||||||
ReadTimeout: envDuration("WARPBOX_READ_TIMEOUT", 0),
|
ReadTimeout: envDuration("WARPBOX_READ_TIMEOUT", 0),
|
||||||
WriteTimeout: envDuration("WARPBOX_WRITE_TIMEOUT", 0),
|
WriteTimeout: envDuration("WARPBOX_WRITE_TIMEOUT", 0),
|
||||||
IdleTimeout: envDuration("WARPBOX_IDLE_TIMEOUT", 120*time.Second),
|
IdleTimeout: envDuration("WARPBOX_IDLE_TIMEOUT", 120*time.Second),
|
||||||
TrustedProxies: envCSV("WARPBOX_TRUSTED_PROXIES"),
|
TrustedProxies: envCSV("WARPBOX_TRUSTED_PROXIES"),
|
||||||
JobsEnabled: envBool("WARPBOX_JOBS_ENABLED", true),
|
JobsEnabled: envBool("WARPBOX_JOBS_ENABLED", true),
|
||||||
CleanupEnabled: envBool("WARPBOX_CLEANUP_ENABLED", true),
|
CleanupEnabled: envBool("WARPBOX_CLEANUP_ENABLED", true),
|
||||||
CleanupEvery: envDuration("WARPBOX_CLEANUP_EVERY", time.Hour),
|
CleanupEvery: envDuration("WARPBOX_CLEANUP_EVERY", time.Hour),
|
||||||
ThumbnailEnabled: envBool("WARPBOX_THUMBNAIL_ENABLED", true),
|
ThumbnailEnabled: envBool("WARPBOX_THUMBNAIL_ENABLED", true),
|
||||||
ThumbnailEvery: envDuration("WARPBOX_THUMBNAIL_EVERY", time.Minute),
|
ThumbnailEvery: envDuration("WARPBOX_THUMBNAIL_EVERY", time.Minute),
|
||||||
MaxUploadSize: envMegabytes("WARPBOX_MAX_UPLOAD_SIZE_MB", 2048), // 2 GiB default.
|
ResumableUploadsEnabled: envBool("WARPBOX_RESUMABLE_UPLOADS_ENABLED", true),
|
||||||
|
ResumableChunkSize: envMegabytes("WARPBOX_RESUMABLE_CHUNK_MB", 8),
|
||||||
|
ResumableRetention: time.Duration(envInt("WARPBOX_RESUMABLE_RETENTION_HOURS", 24)) * time.Hour,
|
||||||
|
MaxUploadSize: envMegabytes("WARPBOX_MAX_UPLOAD_SIZE_MB", 2048), // 2 GiB default.
|
||||||
DefaultSettings: SettingsDefaults{
|
DefaultSettings: SettingsDefaults{
|
||||||
AnonymousUploadsEnabled: envBool("WARPBOX_ANONYMOUS_UPLOADS_ENABLED", true),
|
AnonymousUploadsEnabled: envBool("WARPBOX_ANONYMOUS_UPLOADS_ENABLED", true),
|
||||||
AnonymousMaxUploadMB: envMegabytesLimitFloat("WARPBOX_ANONYMOUS_MAX_UPLOAD_MB", 512),
|
AnonymousMaxUploadMB: envMegabytesLimitFloat("WARPBOX_ANONYMOUS_MAX_UPLOAD_MB", 512),
|
||||||
@@ -103,6 +109,12 @@ func Load() (Config, error) {
|
|||||||
if cfg.MaxUploadSize <= 0 {
|
if cfg.MaxUploadSize <= 0 {
|
||||||
return Config{}, fmt.Errorf("WARPBOX_MAX_UPLOAD_SIZE_MB must be positive")
|
return Config{}, fmt.Errorf("WARPBOX_MAX_UPLOAD_SIZE_MB must be positive")
|
||||||
}
|
}
|
||||||
|
if cfg.ResumableChunkSize <= 0 {
|
||||||
|
return Config{}, fmt.Errorf("WARPBOX_RESUMABLE_CHUNK_MB must be positive")
|
||||||
|
}
|
||||||
|
if cfg.ResumableRetention <= 0 {
|
||||||
|
return Config{}, fmt.Errorf("WARPBOX_RESUMABLE_RETENTION_HOURS must be positive")
|
||||||
|
}
|
||||||
if !validUnlimitedMegabyteLimit(cfg.DefaultSettings.AnonymousMaxUploadMB) ||
|
if !validUnlimitedMegabyteLimit(cfg.DefaultSettings.AnonymousMaxUploadMB) ||
|
||||||
!validUnlimitedMegabyteLimit(cfg.DefaultSettings.AnonymousDailyUploadMB) ||
|
!validUnlimitedMegabyteLimit(cfg.DefaultSettings.AnonymousDailyUploadMB) ||
|
||||||
!validUnlimitedMegabyteLimit(cfg.DefaultSettings.UserDailyUploadMB) ||
|
!validUnlimitedMegabyteLimit(cfg.DefaultSettings.UserDailyUploadMB) ||
|
||||||
|
|||||||
@@ -68,4 +68,13 @@ func TestLoadDefaultsUseLargeUploadFriendlyTimeouts(t *testing.T) {
|
|||||||
if cfg.WriteTimeout != 0 {
|
if cfg.WriteTimeout != 0 {
|
||||||
t.Fatalf("WriteTimeout = %s, want 0 for long uploads", cfg.WriteTimeout)
|
t.Fatalf("WriteTimeout = %s, want 0 for long uploads", cfg.WriteTimeout)
|
||||||
}
|
}
|
||||||
|
if !cfg.ResumableUploadsEnabled {
|
||||||
|
t.Fatalf("ResumableUploadsEnabled = false, want true")
|
||||||
|
}
|
||||||
|
if cfg.ResumableChunkSize != 8*1024*1024 {
|
||||||
|
t.Fatalf("ResumableChunkSize = %d, want 8 MiB", cfg.ResumableChunkSize)
|
||||||
|
}
|
||||||
|
if cfg.ResumableRetention != 24*time.Hour {
|
||||||
|
t.Fatalf("ResumableRetention = %s, want 24h", cfg.ResumableRetention)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -141,6 +141,12 @@ func (a *App) RegisterRoutes(mux *http.ServeMux) {
|
|||||||
mux.HandleFunc("GET /api/v1/schemas/upload-request.json", a.UploadRequestSchema)
|
mux.HandleFunc("GET /api/v1/schemas/upload-request.json", a.UploadRequestSchema)
|
||||||
mux.HandleFunc("GET /api/v1/schemas/upload-response.json", a.UploadResponseSchema)
|
mux.HandleFunc("GET /api/v1/schemas/upload-response.json", a.UploadResponseSchema)
|
||||||
mux.HandleFunc("POST /api/v1/upload", a.Upload)
|
mux.HandleFunc("POST /api/v1/upload", a.Upload)
|
||||||
|
mux.HandleFunc("POST /api/v1/uploads/resumable", a.CreateResumableUpload)
|
||||||
|
mux.HandleFunc("GET /api/v1/uploads/resumable/{sessionID}", a.ResumableUploadStatus)
|
||||||
|
mux.HandleFunc("POST /api/v1/uploads/resumable/{sessionID}/files", a.AddResumableFiles)
|
||||||
|
mux.HandleFunc("PUT /api/v1/uploads/resumable/{sessionID}/files/{fileID}/chunks/{index}", a.PutResumableChunk)
|
||||||
|
mux.HandleFunc("POST /api/v1/uploads/resumable/{sessionID}/complete", a.CompleteResumableUpload)
|
||||||
|
mux.HandleFunc("DELETE /api/v1/uploads/resumable/{sessionID}", a.CancelResumableUpload)
|
||||||
mux.HandleFunc("GET /emoji/{pack}/{file}", a.EmojiAsset)
|
mux.HandleFunc("GET /emoji/{pack}/{file}", a.EmojiAsset)
|
||||||
mux.Handle("GET /static/", a.Static())
|
mux.Handle("GET /static/", a.Static())
|
||||||
}
|
}
|
||||||
|
|||||||
328
backend/libs/handlers/resumable.go
Normal file
328
backend/libs/handlers/resumable.go
Normal file
@@ -0,0 +1,328 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"warpbox.dev/backend/libs/helpers"
|
||||||
|
"warpbox.dev/backend/libs/jobs"
|
||||||
|
"warpbox.dev/backend/libs/services"
|
||||||
|
)
|
||||||
|
|
||||||
|
type resumableCreateRequest struct {
|
||||||
|
Files []services.ResumableFileInput `json:"files"`
|
||||||
|
MaxDays int `json:"maxDays"`
|
||||||
|
ExpiresMinutes int `json:"expiresMinutes"`
|
||||||
|
MaxDownloads int `json:"maxDownloads"`
|
||||||
|
Password string `json:"password"`
|
||||||
|
ObfuscateMetadata bool `json:"obfuscateMetadata"`
|
||||||
|
CollectionID string `json:"collectionId"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type resumableSessionResponse struct {
|
||||||
|
SessionID string `json:"sessionId"`
|
||||||
|
ChunkSize int64 `json:"chunkSize"`
|
||||||
|
Status string `json:"status"`
|
||||||
|
BoxID string `json:"boxId,omitempty"`
|
||||||
|
ExpiresAt string `json:"expiresAt"`
|
||||||
|
Files []services.ResumableFile `json:"files"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *App) CreateResumableUpload(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if !a.cfg.ResumableUploadsEnabled {
|
||||||
|
helpers.WriteJSONError(w, http.StatusForbidden, "resumable uploads are disabled")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
user, loggedIn, authErr := a.currentUserWithAuthError(r)
|
||||||
|
if authErr != nil {
|
||||||
|
a.logger.Warn("resumable upload rejected invalid bearer token", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4011)...)
|
||||||
|
helpers.WriteJSONError(w, http.StatusUnauthorized, "invalid bearer token")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
isAdminUpload := loggedIn && user.Role == services.UserRoleAdmin
|
||||||
|
settings, policy, ok := a.loadUploadPolicyForAPI(w, r, user, loggedIn)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !loggedIn && !settings.AnonymousUploadsEnabled {
|
||||||
|
a.logger.Warn("resumable anonymous upload rejected disabled", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4013)...)
|
||||||
|
helpers.WriteJSONError(w, http.StatusForbidden, "anonymous uploads are disabled")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
rateKey := uploadRateKey(r, user, loggedIn)
|
||||||
|
if !isAdminUpload && policy.ShortRequests > 0 && !a.rateLimiter.Allow("upload:"+rateKey, policy.ShortRequests, policy.ShortWindow, time.Now().UTC()) {
|
||||||
|
a.logger.Warn("resumable upload rate limited", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4291, "user_id", user.ID)...)
|
||||||
|
helpers.WriteJSONError(w, http.StatusTooManyRequests, "too many upload requests, please slow down")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
var payload resumableCreateRequest
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
|
||||||
|
helpers.WriteJSONError(w, http.StatusBadRequest, "upload session request could not be read")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fileSizes := make([]int64, 0, len(payload.Files))
|
||||||
|
var totalBytes int64
|
||||||
|
for _, file := range payload.Files {
|
||||||
|
fileSizes = append(fileSizes, file.Size)
|
||||||
|
totalBytes += file.Size
|
||||||
|
}
|
||||||
|
if !isAdminUpload {
|
||||||
|
if status, message := a.checkUploadPolicyForSizes(r, user, loggedIn, settings, policy, fileSizes, totalBytes); message != "" {
|
||||||
|
a.logger.Warn("resumable upload rejected by policy", withRequestLogAttrs(r, "source", "quota", "severity", "warn", "code", status, "user_id", user.ID, "message", message, "bytes", totalBytes, "files", len(payload.Files))...)
|
||||||
|
helpers.WriteJSONError(w, status, message)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if status, message := a.checkBoxCreationPolicy(r, user, loggedIn, policy); message != "" {
|
||||||
|
a.logger.Warn("resumable upload rejected by box policy", withRequestLogAttrs(r, "source", "quota", "severity", "warn", "code", status, "user_id", user.ID, "message", message, "bytes", totalBytes, "files", len(payload.Files))...)
|
||||||
|
helpers.WriteJSONError(w, status, message)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
opts, err := a.resumableUploadOptions(r, payload, user, loggedIn, isAdminUpload, policy)
|
||||||
|
if err != nil {
|
||||||
|
helpers.WriteJSONError(w, http.StatusRequestEntityTooLarge, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
session, err := a.uploadService.CreateResumableSession(payload.Files, opts, a.cfg.ResumableChunkSize, a.cfg.ResumableRetention)
|
||||||
|
if err != nil {
|
||||||
|
a.logger.Warn("resumable session create failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4002, "user_id", user.ID, "error", err.Error())...)
|
||||||
|
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a.logger.Info("resumable upload session created", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2002, "user_id", user.ID, "session_id", session.ID, "files", len(session.Files), "bytes", totalBytes, "anonymous", !loggedIn)...)
|
||||||
|
helpers.WriteJSON(w, http.StatusCreated, resumableResponse(session))
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *App) ResumableUploadStatus(w http.ResponseWriter, r *http.Request) {
|
||||||
|
session, ok := a.authorizedResumableSession(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
helpers.WriteJSON(w, http.StatusOK, resumableResponse(session))
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *App) AddResumableFiles(w http.ResponseWriter, r *http.Request) {
|
||||||
|
session, ok := a.authorizedResumableSession(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
user, loggedIn, _ := a.currentUserWithAuthError(r)
|
||||||
|
isAdminUpload := loggedIn && user.Role == services.UserRoleAdmin
|
||||||
|
settings, policy, ok := a.loadUploadPolicyForAPI(w, r, user, loggedIn)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var payload struct {
|
||||||
|
Files []services.ResumableFileInput `json:"files"`
|
||||||
|
}
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&payload); err != nil {
|
||||||
|
helpers.WriteJSONError(w, http.StatusBadRequest, "upload files request could not be read")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fileSizes := make([]int64, 0, len(session.Files)+len(payload.Files))
|
||||||
|
var totalBytes int64
|
||||||
|
for _, file := range session.Files {
|
||||||
|
fileSizes = append(fileSizes, file.Size)
|
||||||
|
totalBytes += file.Size
|
||||||
|
}
|
||||||
|
for _, file := range payload.Files {
|
||||||
|
fileSizes = append(fileSizes, file.Size)
|
||||||
|
totalBytes += file.Size
|
||||||
|
}
|
||||||
|
if !isAdminUpload {
|
||||||
|
if status, message := a.checkUploadPolicyForSizes(r, user, loggedIn, settings, policy, fileSizes, totalBytes); message != "" {
|
||||||
|
helpers.WriteJSONError(w, status, message)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
updated, err := a.uploadService.AddResumableFiles(session.ID, payload.Files)
|
||||||
|
if err != nil {
|
||||||
|
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a.logger.Info("resumable upload files added", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2006, "session_id", session.ID, "added", len(updated.Files)-len(session.Files), "files", len(updated.Files))...)
|
||||||
|
helpers.WriteJSON(w, http.StatusOK, resumableResponse(updated))
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *App) PutResumableChunk(w http.ResponseWriter, r *http.Request) {
|
||||||
|
session, ok := a.authorizedResumableSession(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fileID := r.PathValue("fileID")
|
||||||
|
index, err := strconv.Atoi(r.PathValue("index"))
|
||||||
|
if err != nil {
|
||||||
|
helpers.WriteJSONError(w, http.StatusBadRequest, "chunk index is invalid")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
updated, err := a.uploadService.PutResumableChunk(r.Context(), session.ID, fileID, index, r.Body)
|
||||||
|
if err != nil {
|
||||||
|
a.logger.Warn("resumable chunk failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4003, "session_id", session.ID, "file_id", fileID, "chunk", index, "error", err.Error())...)
|
||||||
|
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a.logger.Info("resumable chunk uploaded", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2003, "session_id", session.ID, "file_id", fileID, "chunk", index)...)
|
||||||
|
helpers.WriteJSON(w, http.StatusOK, resumableResponse(updated))
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *App) CompleteResumableUpload(w http.ResponseWriter, r *http.Request) {
|
||||||
|
session, ok := a.authorizedResumableSession(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
user, loggedIn, _ := a.currentUserWithAuthError(r)
|
||||||
|
isAdminUpload := loggedIn && user.Role == services.UserRoleAdmin
|
||||||
|
settings, policy, ok := a.loadUploadPolicyForAPI(w, r, user, loggedIn)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fileSizes := make([]int64, 0, len(session.Files))
|
||||||
|
var totalBytes int64
|
||||||
|
for _, file := range session.Files {
|
||||||
|
fileSizes = append(fileSizes, file.Size)
|
||||||
|
totalBytes += file.Size
|
||||||
|
}
|
||||||
|
if !isAdminUpload {
|
||||||
|
if status, message := a.checkUploadPolicyForSizes(r, user, loggedIn, settings, policy, fileSizes, totalBytes); message != "" {
|
||||||
|
helpers.WriteJSONError(w, status, message)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if status, message := a.checkBoxCreationPolicy(r, user, loggedIn, policy); message != "" {
|
||||||
|
helpers.WriteJSONError(w, status, message)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if status, message := a.checkStorageBackendCapacity(session.Options.StorageBackendID, settings, totalBytes); message != "" {
|
||||||
|
helpers.WriteJSONError(w, status, message)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result, completed, err := a.uploadService.CompleteResumableSession(r.Context(), session.ID)
|
||||||
|
if err != nil {
|
||||||
|
a.logger.Warn("resumable upload complete failed", withRequestLogAttrs(r, "source", "user-upload", "severity", "warn", "code", 4004, "session_id", session.ID, "error", err.Error())...)
|
||||||
|
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !isAdminUpload {
|
||||||
|
if err := a.recordUploadUsage(r, user, loggedIn, totalBytes, 1); err != nil {
|
||||||
|
a.logger.Warn("failed to record resumable upload usage", "source", "quota", "severity", "warn", "code", 4404, "error", err.Error())
|
||||||
|
}
|
||||||
|
if err := a.settingsService.CleanupUsage(time.Now().UTC(), settings.UsageRetentionDays); err != nil {
|
||||||
|
a.logger.Warn("failed to cleanup upload usage", "source", "quota", "severity", "warn", "code", 4405, "error", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
jobs.GenerateThumbnailsForBoxAsync(a.uploadService, a.logger, result.BoxID)
|
||||||
|
a.logger.Info("resumable upload completed", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2004, "user_id", user.ID, "session_id", completed.ID, "box_id", result.BoxID, "files", len(result.Files), "bytes", totalBytes, "admin", isAdminUpload, "anonymous", !loggedIn)...)
|
||||||
|
helpers.WriteJSON(w, http.StatusCreated, result)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *App) CancelResumableUpload(w http.ResponseWriter, r *http.Request) {
|
||||||
|
session, ok := a.authorizedResumableSession(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := a.uploadService.CancelResumableSession(session.ID); err != nil {
|
||||||
|
helpers.WriteJSONError(w, http.StatusBadRequest, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a.logger.Info("resumable upload cancelled", withRequestLogAttrs(r, "source", "user-upload", "severity", "user_activity", "code", 2005, "session_id", session.ID)...)
|
||||||
|
w.WriteHeader(http.StatusNoContent)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *App) authorizedResumableSession(w http.ResponseWriter, r *http.Request) (services.ResumableSession, bool) {
|
||||||
|
user, loggedIn, authErr := a.currentUserWithAuthError(r)
|
||||||
|
if authErr != nil {
|
||||||
|
helpers.WriteJSONError(w, http.StatusUnauthorized, "invalid bearer token")
|
||||||
|
return services.ResumableSession{}, false
|
||||||
|
}
|
||||||
|
session, err := a.uploadService.GetResumableSession(r.PathValue("sessionID"))
|
||||||
|
if err != nil {
|
||||||
|
helpers.WriteJSONError(w, http.StatusNotFound, "upload session not found")
|
||||||
|
return services.ResumableSession{}, false
|
||||||
|
}
|
||||||
|
if loggedIn {
|
||||||
|
if session.Options.OwnerID != user.ID {
|
||||||
|
helpers.WriteJSONError(w, http.StatusForbidden, "upload session not found")
|
||||||
|
return services.ResumableSession{}, false
|
||||||
|
}
|
||||||
|
return session, true
|
||||||
|
}
|
||||||
|
if session.Options.OwnerID != "" || session.Options.CreatorIP != uploadClientIP(r) {
|
||||||
|
helpers.WriteJSONError(w, http.StatusForbidden, "upload session not found")
|
||||||
|
return services.ResumableSession{}, false
|
||||||
|
}
|
||||||
|
return session, true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *App) loadUploadPolicyForAPI(w http.ResponseWriter, r *http.Request, user services.User, loggedIn bool) (services.UploadPolicySettings, services.EffectiveUploadPolicy, bool) {
|
||||||
|
settings, err := a.settingsService.UploadPolicy()
|
||||||
|
if err != nil {
|
||||||
|
a.logger.Error("failed to load upload policy", "source", "settings", "severity", "error", "code", 5006, "error", err.Error())
|
||||||
|
helpers.WriteJSONError(w, http.StatusInternalServerError, "upload policy could not be loaded")
|
||||||
|
return services.UploadPolicySettings{}, services.EffectiveUploadPolicy{}, false
|
||||||
|
}
|
||||||
|
return settings, a.effectiveUploadPolicy(settings, user, loggedIn), true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *App) resumableUploadOptions(r *http.Request, payload resumableCreateRequest, user services.User, loggedIn, isAdminUpload bool, policy services.EffectiveUploadPolicy) (services.UploadOptions, error) {
|
||||||
|
var ownerID string
|
||||||
|
var collectionID string
|
||||||
|
if loggedIn {
|
||||||
|
ownerID = user.ID
|
||||||
|
collectionID = strings.TrimSpace(payload.CollectionID)
|
||||||
|
if !a.authService.CollectionOwnedBy(collectionID, user.ID) {
|
||||||
|
return services.UploadOptions{}, fmt.Errorf("collection not found")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
unlimitedExpiry := isAdminUpload || policy.MaxDays < 0
|
||||||
|
rawMaxDays := payload.MaxDays
|
||||||
|
maxDays := rawMaxDays
|
||||||
|
if maxDays <= 0 {
|
||||||
|
maxDays = 7
|
||||||
|
if policy.MaxDays > 0 && policy.MaxDays < maxDays {
|
||||||
|
maxDays = policy.MaxDays
|
||||||
|
}
|
||||||
|
}
|
||||||
|
expiresMinutes := payload.ExpiresMinutes
|
||||||
|
if expiresMinutes < 0 || rawMaxDays < 0 {
|
||||||
|
if !unlimitedExpiry {
|
||||||
|
return services.UploadOptions{}, fmt.Errorf("expiration cannot exceed %d days", policy.MaxDays)
|
||||||
|
}
|
||||||
|
expiresMinutes = -1
|
||||||
|
} else if !unlimitedExpiry {
|
||||||
|
if maxDays > policy.MaxDays {
|
||||||
|
return services.UploadOptions{}, fmt.Errorf("expiration cannot exceed %d days", policy.MaxDays)
|
||||||
|
}
|
||||||
|
if expiresMinutes > 0 && expiresMinutes > policy.MaxDays*24*60 {
|
||||||
|
return services.UploadOptions{}, fmt.Errorf("expiration cannot exceed %d days", policy.MaxDays)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return services.UploadOptions{
|
||||||
|
MaxDays: maxDays,
|
||||||
|
ExpiresInMinutes: expiresMinutes,
|
||||||
|
MaxDownloads: payload.MaxDownloads,
|
||||||
|
Password: payload.Password,
|
||||||
|
ObfuscateMetadata: payload.ObfuscateMetadata,
|
||||||
|
OwnerID: ownerID,
|
||||||
|
CollectionID: collectionID,
|
||||||
|
SkipSizeLimit: isAdminUpload || policy.MaxUploadMB < 0,
|
||||||
|
CreatorIP: uploadClientIP(r),
|
||||||
|
StorageBackendID: policy.StorageBackendID,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resumableResponse(session services.ResumableSession) resumableSessionResponse {
|
||||||
|
return resumableSessionResponse{
|
||||||
|
SessionID: session.ID,
|
||||||
|
ChunkSize: session.ChunkSize,
|
||||||
|
Status: session.Status,
|
||||||
|
BoxID: session.BoxID,
|
||||||
|
ExpiresAt: session.ExpiresAt.Format(time.RFC3339),
|
||||||
|
Files: session.Files,
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -228,11 +228,22 @@ func (a *App) checkUploadPolicy(r *http.Request, user services.User, loggedIn bo
|
|||||||
if len(files) == 0 {
|
if len(files) == 0 {
|
||||||
return 0, ""
|
return 0, ""
|
||||||
}
|
}
|
||||||
|
sizes := make([]int64, 0, len(files))
|
||||||
|
for _, file := range files {
|
||||||
|
sizes = append(sizes, file.Size)
|
||||||
|
}
|
||||||
|
return a.checkUploadPolicyForSizes(r, user, loggedIn, settings, policy, sizes, totalBytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *App) checkUploadPolicyForSizes(r *http.Request, user services.User, loggedIn bool, settings services.UploadPolicySettings, policy services.EffectiveUploadPolicy, fileSizes []int64, totalBytes int64) (int, string) {
|
||||||
|
if len(fileSizes) == 0 {
|
||||||
|
return 0, ""
|
||||||
|
}
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
if policy.MaxUploadMB > 0 {
|
if policy.MaxUploadMB > 0 {
|
||||||
maxBytes := services.MegabytesToBytes(policy.MaxUploadMB)
|
maxBytes := services.MegabytesToBytes(policy.MaxUploadMB)
|
||||||
for _, file := range files {
|
for _, fileSize := range fileSizes {
|
||||||
if file.Size > maxBytes {
|
if fileSize > maxBytes {
|
||||||
return http.StatusRequestEntityTooLarge, "file exceeds upload size limit"
|
return http.StatusRequestEntityTooLarge, "file exceeds upload size limit"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package handlers
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
@@ -10,8 +11,10 @@ import (
|
|||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"warpbox.dev/backend/libs/config"
|
"warpbox.dev/backend/libs/config"
|
||||||
"warpbox.dev/backend/libs/services"
|
"warpbox.dev/backend/libs/services"
|
||||||
@@ -103,6 +106,196 @@ func TestUploadTextResponseReturnsOnlyBoxURL(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestResumableUploadFlowCreatesNormalBox(t *testing.T) {
|
||||||
|
app, cleanup := newTestApp(t)
|
||||||
|
defer cleanup()
|
||||||
|
|
||||||
|
createBody := `{"files":[{"name":"note.txt","size":11,"contentType":"text/plain"}],"expiresMinutes":60}`
|
||||||
|
createRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable", strings.NewReader(createBody))
|
||||||
|
createRequest.Header.Set("Accept", "application/json")
|
||||||
|
createResponse := httptest.NewRecorder()
|
||||||
|
app.CreateResumableUpload(createResponse, createRequest)
|
||||||
|
if createResponse.Code != http.StatusCreated {
|
||||||
|
t.Fatalf("create status = %d, body = %s", createResponse.Code, createResponse.Body.String())
|
||||||
|
}
|
||||||
|
var session struct {
|
||||||
|
SessionID string `json:"sessionId"`
|
||||||
|
ChunkSize int64 `json:"chunkSize"`
|
||||||
|
Files []struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
ChunkCount int `json:"chunkCount"`
|
||||||
|
UploadedChunks []int `json:"uploadedChunks"`
|
||||||
|
} `json:"files"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(createResponse.Body.Bytes(), &session); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal session returned error: %v", err)
|
||||||
|
}
|
||||||
|
if session.SessionID == "" || session.ChunkSize != 4 || len(session.Files) != 1 || session.Files[0].ChunkCount != 3 {
|
||||||
|
t.Fatalf("unexpected session response: %+v", session)
|
||||||
|
}
|
||||||
|
|
||||||
|
chunks := map[int]string{1: "o wo", 0: "hell", 2: "rld"}
|
||||||
|
for index, body := range chunks {
|
||||||
|
request := httptest.NewRequest(http.MethodPut, "/api/v1/uploads/resumable/"+session.SessionID+"/files/"+session.Files[0].ID+"/chunks/"+strconv.Itoa(index), strings.NewReader(body))
|
||||||
|
request.SetPathValue("sessionID", session.SessionID)
|
||||||
|
request.SetPathValue("fileID", session.Files[0].ID)
|
||||||
|
request.SetPathValue("index", strconv.Itoa(index))
|
||||||
|
response := httptest.NewRecorder()
|
||||||
|
app.PutResumableChunk(response, request)
|
||||||
|
if response.Code != http.StatusOK {
|
||||||
|
t.Fatalf("chunk %d status = %d, body = %s", index, response.Code, response.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
completeRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
|
||||||
|
completeRequest.SetPathValue("sessionID", session.SessionID)
|
||||||
|
completeResponse := httptest.NewRecorder()
|
||||||
|
app.CompleteResumableUpload(completeResponse, completeRequest)
|
||||||
|
if completeResponse.Code != http.StatusCreated {
|
||||||
|
t.Fatalf("complete status = %d, body = %s", completeResponse.Code, completeResponse.Body.String())
|
||||||
|
}
|
||||||
|
var payload services.UploadResult
|
||||||
|
if err := json.Unmarshal(completeResponse.Body.Bytes(), &payload); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal result returned error: %v", err)
|
||||||
|
}
|
||||||
|
box, err := app.uploadService.GetBox(payload.BoxID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetBox returned error: %v", err)
|
||||||
|
}
|
||||||
|
if len(box.Files) != 1 || box.Files[0].Name != "note.txt" || box.Files[0].Size != 11 {
|
||||||
|
t.Fatalf("unexpected box files: %+v", box.Files)
|
||||||
|
}
|
||||||
|
object, err := app.uploadService.OpenFileObject(context.Background(), box, box.Files[0])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("OpenFileObject returned error: %v", err)
|
||||||
|
}
|
||||||
|
data, err := io.ReadAll(object.Body)
|
||||||
|
object.Body.Close()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadAll returned error: %v", err)
|
||||||
|
}
|
||||||
|
if string(data) != "hello world" {
|
||||||
|
t.Fatalf("uploaded body = %q", string(data))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResumableUploadRequiresAllChunks(t *testing.T) {
|
||||||
|
app, cleanup := newTestApp(t)
|
||||||
|
defer cleanup()
|
||||||
|
|
||||||
|
createRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable", strings.NewReader(`{"files":[{"name":"note.txt","size":8,"contentType":"text/plain"}]}`))
|
||||||
|
createResponse := httptest.NewRecorder()
|
||||||
|
app.CreateResumableUpload(createResponse, createRequest)
|
||||||
|
if createResponse.Code != http.StatusCreated {
|
||||||
|
t.Fatalf("create status = %d, body = %s", createResponse.Code, createResponse.Body.String())
|
||||||
|
}
|
||||||
|
var session struct {
|
||||||
|
SessionID string `json:"sessionId"`
|
||||||
|
Files []struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
} `json:"files"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(createResponse.Body.Bytes(), &session); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal session returned error: %v", err)
|
||||||
|
}
|
||||||
|
chunkRequest := httptest.NewRequest(http.MethodPut, "/api/v1/uploads/resumable/"+session.SessionID+"/files/"+session.Files[0].ID+"/chunks/0", strings.NewReader("hell"))
|
||||||
|
chunkRequest.SetPathValue("sessionID", session.SessionID)
|
||||||
|
chunkRequest.SetPathValue("fileID", session.Files[0].ID)
|
||||||
|
chunkRequest.SetPathValue("index", "0")
|
||||||
|
chunkResponse := httptest.NewRecorder()
|
||||||
|
app.PutResumableChunk(chunkResponse, chunkRequest)
|
||||||
|
if chunkResponse.Code != http.StatusOK {
|
||||||
|
t.Fatalf("chunk status = %d, body = %s", chunkResponse.Code, chunkResponse.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
completeRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
|
||||||
|
completeRequest.SetPathValue("sessionID", session.SessionID)
|
||||||
|
completeResponse := httptest.NewRecorder()
|
||||||
|
app.CompleteResumableUpload(completeResponse, completeRequest)
|
||||||
|
if completeResponse.Code != http.StatusBadRequest {
|
||||||
|
t.Fatalf("complete status = %d, body = %s", completeResponse.Code, completeResponse.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResumableUploadCanAddFilesToExistingSession(t *testing.T) {
|
||||||
|
app, cleanup := newTestApp(t)
|
||||||
|
defer cleanup()
|
||||||
|
|
||||||
|
createBody := `{"files":[{"name":"one.txt","size":4,"contentType":"text/plain","fingerprint":"one"}],"expiresMinutes":60}`
|
||||||
|
createRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable", strings.NewReader(createBody))
|
||||||
|
createResponse := httptest.NewRecorder()
|
||||||
|
app.CreateResumableUpload(createResponse, createRequest)
|
||||||
|
if createResponse.Code != http.StatusCreated {
|
||||||
|
t.Fatalf("create status = %d, body = %s", createResponse.Code, createResponse.Body.String())
|
||||||
|
}
|
||||||
|
var session struct {
|
||||||
|
SessionID string `json:"sessionId"`
|
||||||
|
Files []struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
} `json:"files"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(createResponse.Body.Bytes(), &session); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal session returned error: %v", err)
|
||||||
|
}
|
||||||
|
firstChunk := httptest.NewRequest(http.MethodPut, "/api/v1/uploads/resumable/"+session.SessionID+"/files/"+session.Files[0].ID+"/chunks/0", strings.NewReader("one!"))
|
||||||
|
firstChunk.SetPathValue("sessionID", session.SessionID)
|
||||||
|
firstChunk.SetPathValue("fileID", session.Files[0].ID)
|
||||||
|
firstChunk.SetPathValue("index", "0")
|
||||||
|
firstChunkResponse := httptest.NewRecorder()
|
||||||
|
app.PutResumableChunk(firstChunkResponse, firstChunk)
|
||||||
|
if firstChunkResponse.Code != http.StatusOK {
|
||||||
|
t.Fatalf("first chunk status = %d, body = %s", firstChunkResponse.Code, firstChunkResponse.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
addRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/files", strings.NewReader(`{"files":[{"name":"two.txt","size":4,"contentType":"text/plain","fingerprint":"two"}]}`))
|
||||||
|
addRequest.SetPathValue("sessionID", session.SessionID)
|
||||||
|
addResponse := httptest.NewRecorder()
|
||||||
|
app.AddResumableFiles(addResponse, addRequest)
|
||||||
|
if addResponse.Code != http.StatusOK {
|
||||||
|
t.Fatalf("add status = %d, body = %s", addResponse.Code, addResponse.Body.String())
|
||||||
|
}
|
||||||
|
var updated struct {
|
||||||
|
Files []struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Name string `json:"name"`
|
||||||
|
UploadedChunks []int `json:"uploadedChunks"`
|
||||||
|
} `json:"files"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(addResponse.Body.Bytes(), &updated); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal updated returned error: %v", err)
|
||||||
|
}
|
||||||
|
if len(updated.Files) != 2 || len(updated.Files[0].UploadedChunks) != 1 {
|
||||||
|
t.Fatalf("unexpected updated session: %+v", updated)
|
||||||
|
}
|
||||||
|
secondChunk := httptest.NewRequest(http.MethodPut, "/api/v1/uploads/resumable/"+session.SessionID+"/files/"+updated.Files[1].ID+"/chunks/0", strings.NewReader("two!"))
|
||||||
|
secondChunk.SetPathValue("sessionID", session.SessionID)
|
||||||
|
secondChunk.SetPathValue("fileID", updated.Files[1].ID)
|
||||||
|
secondChunk.SetPathValue("index", "0")
|
||||||
|
secondChunkResponse := httptest.NewRecorder()
|
||||||
|
app.PutResumableChunk(secondChunkResponse, secondChunk)
|
||||||
|
if secondChunkResponse.Code != http.StatusOK {
|
||||||
|
t.Fatalf("second chunk status = %d, body = %s", secondChunkResponse.Code, secondChunkResponse.Body.String())
|
||||||
|
}
|
||||||
|
completeRequest := httptest.NewRequest(http.MethodPost, "/api/v1/uploads/resumable/"+session.SessionID+"/complete", nil)
|
||||||
|
completeRequest.SetPathValue("sessionID", session.SessionID)
|
||||||
|
completeResponse := httptest.NewRecorder()
|
||||||
|
app.CompleteResumableUpload(completeResponse, completeRequest)
|
||||||
|
if completeResponse.Code != http.StatusCreated {
|
||||||
|
t.Fatalf("complete status = %d, body = %s", completeResponse.Code, completeResponse.Body.String())
|
||||||
|
}
|
||||||
|
var payload services.UploadResult
|
||||||
|
if err := json.Unmarshal(completeResponse.Body.Bytes(), &payload); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal result returned error: %v", err)
|
||||||
|
}
|
||||||
|
box, err := app.uploadService.GetBox(payload.BoxID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetBox returned error: %v", err)
|
||||||
|
}
|
||||||
|
if len(box.Files) != 2 {
|
||||||
|
t.Fatalf("box file count = %d, want 2", len(box.Files))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestManageBoxAndDeleteFlow(t *testing.T) {
|
func TestManageBoxAndDeleteFlow(t *testing.T) {
|
||||||
app, cleanup := newTestApp(t)
|
app, cleanup := newTestApp(t)
|
||||||
defer cleanup()
|
defer cleanup()
|
||||||
@@ -214,13 +407,16 @@ func newTestApp(t *testing.T) (*App, func()) {
|
|||||||
|
|
||||||
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
|
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
|
||||||
cfg := config.Config{
|
cfg := config.Config{
|
||||||
AppName: "warpbox.dev",
|
AppName: "warpbox.dev",
|
||||||
AppVersion: "test",
|
AppVersion: "test",
|
||||||
BaseURL: "http://example.test",
|
BaseURL: "http://example.test",
|
||||||
DataDir: filepath.Join(root, "data"),
|
DataDir: filepath.Join(root, "data"),
|
||||||
StaticDir: staticDir,
|
StaticDir: staticDir,
|
||||||
TemplateDir: templateDir,
|
TemplateDir: templateDir,
|
||||||
MaxUploadSize: 1024 * 1024,
|
MaxUploadSize: 1024 * 1024,
|
||||||
|
ResumableUploadsEnabled: true,
|
||||||
|
ResumableChunkSize: 4,
|
||||||
|
ResumableRetention: time.Hour,
|
||||||
DefaultSettings: config.SettingsDefaults{
|
DefaultSettings: config.SettingsDefaults{
|
||||||
AnonymousUploadsEnabled: true,
|
AnonymousUploadsEnabled: true,
|
||||||
AnonymousMaxUploadMB: 1,
|
AnonymousMaxUploadMB: 1,
|
||||||
|
|||||||
@@ -22,6 +22,14 @@ func newCleanupJob(cfg config.Config, logger *slog.Logger, uploadService *servic
|
|||||||
if cleaned > 0 {
|
if cleaned > 0 {
|
||||||
logger.Info("cleanup job complete", "source", "housekeeping", "severity", "user_activity", "code", 2202, "cleaned", cleaned)
|
logger.Info("cleanup job complete", "source", "housekeeping", "severity", "user_activity", "code", 2202, "cleaned", cleaned)
|
||||||
}
|
}
|
||||||
|
cleanedUploads, err := uploadService.CleanupExpiredResumableSessions(time.Now().UTC())
|
||||||
|
if err != nil {
|
||||||
|
logger.Warn("resumable upload cleanup failed", "source", "housekeeping", "severity", "warn", "code", 4204, "error", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if cleanedUploads > 0 {
|
||||||
|
logger.Info("resumable uploads cleaned", "source", "housekeeping", "severity", "user_activity", "code", 2204, "cleaned", cleanedUploads)
|
||||||
|
}
|
||||||
if banService != nil {
|
if banService != nil {
|
||||||
cleanedEvents, err := banService.CleanupAbuseEvents(time.Now().UTC())
|
cleanedEvents, err := banService.CleanupAbuseEvents(time.Now().UTC())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -37,7 +45,12 @@ func newCleanupJob(cfg config.Config, logger *slog.Logger, uploadService *servic
|
|||||||
}
|
}
|
||||||
|
|
||||||
func RunCleanupNow(uploadService *services.UploadService, logger *slog.Logger) (int, error) {
|
func RunCleanupNow(uploadService *services.UploadService, logger *slog.Logger) (int, error) {
|
||||||
return cleanupUnavailableBoxes(uploadService, logger)
|
cleaned, err := cleanupUnavailableBoxes(uploadService, logger)
|
||||||
|
if err != nil {
|
||||||
|
return cleaned, err
|
||||||
|
}
|
||||||
|
cleanedUploads, err := uploadService.CleanupExpiredResumableSessions(time.Now().UTC())
|
||||||
|
return cleaned + cleanedUploads, err
|
||||||
}
|
}
|
||||||
|
|
||||||
func cleanupUnavailableBoxes(uploadService *services.UploadService, logger *slog.Logger) (int, error) {
|
func cleanupUnavailableBoxes(uploadService *services.UploadService, logger *slog.Logger) (int, error) {
|
||||||
|
|||||||
454
backend/libs/services/resumable.go
Normal file
454
backend/libs/services/resumable.go
Normal file
@@ -0,0 +1,454 @@
|
|||||||
|
package services
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"go.etcd.io/bbolt"
|
||||||
|
)
|
||||||
|
|
||||||
|
var resumableUploadsBucket = []byte("resumable_uploads")
|
||||||
|
|
||||||
|
const (
|
||||||
|
ResumableStatusUploading = "uploading"
|
||||||
|
ResumableStatusCompleted = "completed"
|
||||||
|
ResumableStatusCancelled = "cancelled"
|
||||||
|
)
|
||||||
|
|
||||||
|
type ResumableFileInput struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Size int64 `json:"size"`
|
||||||
|
ContentType string `json:"contentType"`
|
||||||
|
Fingerprint string `json:"fingerprint,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ResumableSession struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Options UploadOptions `json:"options"`
|
||||||
|
Files []ResumableFile `json:"files"`
|
||||||
|
ChunkSize int64 `json:"chunkSize"`
|
||||||
|
Status string `json:"status"`
|
||||||
|
BoxID string `json:"boxId,omitempty"`
|
||||||
|
CreatedAt time.Time `json:"createdAt"`
|
||||||
|
UpdatedAt time.Time `json:"updatedAt"`
|
||||||
|
ExpiresAt time.Time `json:"expiresAt"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ResumableFile struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Name string `json:"name"`
|
||||||
|
Size int64 `json:"size"`
|
||||||
|
ContentType string `json:"contentType"`
|
||||||
|
Fingerprint string `json:"fingerprint,omitempty"`
|
||||||
|
ChunkCount int `json:"chunkCount"`
|
||||||
|
UploadedChunks []int `json:"uploadedChunks"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) ensureResumableBucket() error {
|
||||||
|
return s.db.Update(func(tx *bbolt.Tx) error {
|
||||||
|
_, err := tx.CreateBucketIfNotExists(resumableUploadsBucket)
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) CreateResumableSession(files []ResumableFileInput, opts UploadOptions, chunkSize int64, retention time.Duration) (ResumableSession, error) {
|
||||||
|
if len(files) == 0 {
|
||||||
|
return ResumableSession{}, fmt.Errorf("no files were uploaded")
|
||||||
|
}
|
||||||
|
if chunkSize <= 0 {
|
||||||
|
return ResumableSession{}, fmt.Errorf("chunk size must be positive")
|
||||||
|
}
|
||||||
|
if retention <= 0 {
|
||||||
|
return ResumableSession{}, fmt.Errorf("retention must be positive")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(opts.Password) != "" {
|
||||||
|
opts.PasswordSalt, opts.PasswordHash = hashPassword(opts.Password)
|
||||||
|
opts.Password = ""
|
||||||
|
}
|
||||||
|
sessionFiles, err := s.resumableFilesFromInput(files, opts, chunkSize, nil)
|
||||||
|
if err != nil {
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
now := time.Now().UTC()
|
||||||
|
session := ResumableSession{
|
||||||
|
ID: randomID(12),
|
||||||
|
Options: opts,
|
||||||
|
Files: sessionFiles,
|
||||||
|
ChunkSize: chunkSize,
|
||||||
|
Status: ResumableStatusUploading,
|
||||||
|
CreatedAt: now,
|
||||||
|
UpdatedAt: now,
|
||||||
|
ExpiresAt: now.Add(retention),
|
||||||
|
}
|
||||||
|
if err := s.saveResumableSession(session); err != nil {
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
return session, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) AddResumableFiles(sessionID string, files []ResumableFileInput) (ResumableSession, error) {
|
||||||
|
if len(files) == 0 {
|
||||||
|
return s.GetResumableSession(sessionID)
|
||||||
|
}
|
||||||
|
session, err := s.GetResumableSession(sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
if err := resumableSessionWritable(session); err != nil {
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
existing := make(map[string]bool)
|
||||||
|
for _, file := range session.Files {
|
||||||
|
existing[resumableFileKey(file.Name, file.Size, file.Fingerprint)] = true
|
||||||
|
}
|
||||||
|
newFiles, err := s.resumableFilesFromInput(files, session.Options, session.ChunkSize, existing)
|
||||||
|
if err != nil {
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
if len(newFiles) == 0 {
|
||||||
|
return session, nil
|
||||||
|
}
|
||||||
|
session.Files = append(session.Files, newFiles...)
|
||||||
|
session.UpdatedAt = time.Now().UTC()
|
||||||
|
if err := s.saveResumableSession(session); err != nil {
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
return session, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) GetResumableSession(id string) (ResumableSession, error) {
|
||||||
|
var session ResumableSession
|
||||||
|
err := s.db.View(func(tx *bbolt.Tx) error {
|
||||||
|
bucket := tx.Bucket(resumableUploadsBucket)
|
||||||
|
if bucket == nil {
|
||||||
|
return os.ErrNotExist
|
||||||
|
}
|
||||||
|
data := bucket.Get([]byte(id))
|
||||||
|
if data == nil {
|
||||||
|
return os.ErrNotExist
|
||||||
|
}
|
||||||
|
return json.Unmarshal(data, &session)
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
return session, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) PutResumableChunk(ctx context.Context, sessionID, fileID string, index int, body io.Reader) (ResumableSession, error) {
|
||||||
|
session, err := s.GetResumableSession(sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
if err := resumableSessionWritable(session); err != nil {
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
fileIndex := -1
|
||||||
|
for i, file := range session.Files {
|
||||||
|
if file.ID == fileID {
|
||||||
|
fileIndex = i
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if fileIndex < 0 {
|
||||||
|
return ResumableSession{}, os.ErrNotExist
|
||||||
|
}
|
||||||
|
file := session.Files[fileIndex]
|
||||||
|
if index < 0 || index >= file.ChunkCount {
|
||||||
|
return ResumableSession{}, fmt.Errorf("chunk index is invalid")
|
||||||
|
}
|
||||||
|
expectedSize := expectedChunkSize(file.Size, session.ChunkSize, index)
|
||||||
|
chunkDir := s.resumableFileDir(session.ID, file.ID)
|
||||||
|
if err := os.MkdirAll(chunkDir, 0o755); err != nil {
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
chunkPath := s.resumableChunkPath(session.ID, file.ID, index)
|
||||||
|
tempPath := chunkPath + ".tmp"
|
||||||
|
target, err := os.OpenFile(tempPath, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
|
||||||
|
if err != nil {
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
written, copyErr := io.Copy(target, io.LimitReader(body, expectedSize+1))
|
||||||
|
closeErr := target.Close()
|
||||||
|
if copyErr != nil {
|
||||||
|
_ = os.Remove(tempPath)
|
||||||
|
return ResumableSession{}, copyErr
|
||||||
|
}
|
||||||
|
if closeErr != nil {
|
||||||
|
_ = os.Remove(tempPath)
|
||||||
|
return ResumableSession{}, closeErr
|
||||||
|
}
|
||||||
|
if written != expectedSize {
|
||||||
|
_ = os.Remove(tempPath)
|
||||||
|
return ResumableSession{}, fmt.Errorf("chunk size mismatch")
|
||||||
|
}
|
||||||
|
if err := os.Rename(tempPath, chunkPath); err != nil {
|
||||||
|
_ = os.Remove(tempPath)
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
session.Files[fileIndex].UploadedChunks = addChunkIndex(session.Files[fileIndex].UploadedChunks, index)
|
||||||
|
session.UpdatedAt = time.Now().UTC()
|
||||||
|
if err := s.saveResumableSession(session); err != nil {
|
||||||
|
return ResumableSession{}, err
|
||||||
|
}
|
||||||
|
return session, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) CompleteResumableSession(ctx context.Context, sessionID string) (UploadResult, ResumableSession, error) {
|
||||||
|
session, err := s.GetResumableSession(sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
if err := resumableSessionWritable(session); err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
staged, cleanup, err := s.assembleResumableFiles(ctx, session)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
defer cleanup()
|
||||||
|
|
||||||
|
result, err := s.CreateBoxFromIncoming(staged, session.Options)
|
||||||
|
if err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
if err := os.RemoveAll(s.resumableSessionDir(session.ID)); err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
session.Status = ResumableStatusCompleted
|
||||||
|
session.BoxID = result.BoxID
|
||||||
|
session.UpdatedAt = time.Now().UTC()
|
||||||
|
if err := s.saveResumableSession(session); err != nil {
|
||||||
|
return UploadResult{}, ResumableSession{}, err
|
||||||
|
}
|
||||||
|
return result, session, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) CancelResumableSession(sessionID string) error {
|
||||||
|
session, err := s.GetResumableSession(sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
session.Status = ResumableStatusCancelled
|
||||||
|
session.UpdatedAt = time.Now().UTC()
|
||||||
|
if err := s.saveResumableSession(session); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return os.RemoveAll(s.resumableSessionDir(session.ID))
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) CleanupExpiredResumableSessions(now time.Time) (int, error) {
|
||||||
|
candidates := make([]ResumableSession, 0)
|
||||||
|
err := s.db.View(func(tx *bbolt.Tx) error {
|
||||||
|
bucket := tx.Bucket(resumableUploadsBucket)
|
||||||
|
if bucket == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return bucket.ForEach(func(_, value []byte) error {
|
||||||
|
var session ResumableSession
|
||||||
|
if err := json.Unmarshal(value, &session); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !session.ExpiresAt.After(now) || session.Status != ResumableStatusUploading {
|
||||||
|
candidates = append(candidates, session)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
for _, session := range candidates {
|
||||||
|
if err := os.RemoveAll(s.resumableSessionDir(session.ID)); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
err = s.db.Update(func(tx *bbolt.Tx) error {
|
||||||
|
bucket := tx.Bucket(resumableUploadsBucket)
|
||||||
|
if bucket == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
for _, session := range candidates {
|
||||||
|
if err := bucket.Delete([]byte(session.ID)); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
return len(candidates), err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) saveResumableSession(session ResumableSession) error {
|
||||||
|
if err := s.ensureResumableBucket(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return s.db.Update(func(tx *bbolt.Tx) error {
|
||||||
|
data, err := json.Marshal(session)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return tx.Bucket(resumableUploadsBucket).Put([]byte(session.ID), data)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) resumableFilesFromInput(files []ResumableFileInput, opts UploadOptions, chunkSize int64, existing map[string]bool) ([]ResumableFile, error) {
|
||||||
|
sessionFiles := make([]ResumableFile, 0, len(files))
|
||||||
|
for _, file := range files {
|
||||||
|
file.Name = filepath.Base(strings.TrimSpace(file.Name))
|
||||||
|
if file.Name == "." || file.Name == "" {
|
||||||
|
return nil, fmt.Errorf("file name is required")
|
||||||
|
}
|
||||||
|
if file.Size < 0 {
|
||||||
|
return nil, fmt.Errorf("file size is invalid")
|
||||||
|
}
|
||||||
|
fingerprint := strings.TrimSpace(file.Fingerprint)
|
||||||
|
key := resumableFileKey(file.Name, file.Size, fingerprint)
|
||||||
|
if existing != nil && existing[key] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !opts.SkipSizeLimit {
|
||||||
|
if err := s.ValidateSize(file.Size); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
chunks := int((file.Size + chunkSize - 1) / chunkSize)
|
||||||
|
if chunks == 0 {
|
||||||
|
chunks = 1
|
||||||
|
}
|
||||||
|
sessionFiles = append(sessionFiles, ResumableFile{
|
||||||
|
ID: randomID(8),
|
||||||
|
Name: file.Name,
|
||||||
|
Size: file.Size,
|
||||||
|
ContentType: strings.TrimSpace(file.ContentType),
|
||||||
|
Fingerprint: fingerprint,
|
||||||
|
ChunkCount: chunks,
|
||||||
|
})
|
||||||
|
if existing != nil {
|
||||||
|
existing[key] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return sessionFiles, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resumableFileKey(name string, size int64, fingerprint string) string {
|
||||||
|
return strings.TrimSpace(fingerprint) + "|" + filepath.Base(strings.TrimSpace(name)) + "|" + fmt.Sprintf("%d", size)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) assembleResumableFiles(ctx context.Context, session ResumableSession) ([]IncomingFile, func(), error) {
|
||||||
|
assembledDir := filepath.Join(s.resumableSessionDir(session.ID), "assembled")
|
||||||
|
if err := os.MkdirAll(assembledDir, 0o755); err != nil {
|
||||||
|
return nil, func() {}, err
|
||||||
|
}
|
||||||
|
cleanup := func() {
|
||||||
|
_ = os.RemoveAll(assembledDir)
|
||||||
|
}
|
||||||
|
staged := make([]IncomingFile, 0, len(session.Files))
|
||||||
|
for _, file := range session.Files {
|
||||||
|
if len(file.UploadedChunks) != file.ChunkCount {
|
||||||
|
cleanup()
|
||||||
|
return nil, func() {}, fmt.Errorf("file %s is missing chunks", file.Name)
|
||||||
|
}
|
||||||
|
assembledPath := filepath.Join(assembledDir, file.ID)
|
||||||
|
target, err := os.OpenFile(assembledPath, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o600)
|
||||||
|
if err != nil {
|
||||||
|
cleanup()
|
||||||
|
return nil, func() {}, err
|
||||||
|
}
|
||||||
|
var written int64
|
||||||
|
for i := 0; i < file.ChunkCount; i++ {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
_ = target.Close()
|
||||||
|
cleanup()
|
||||||
|
return nil, func() {}, ctx.Err()
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
chunk, err := os.Open(s.resumableChunkPath(session.ID, file.ID, i))
|
||||||
|
if err != nil {
|
||||||
|
_ = target.Close()
|
||||||
|
cleanup()
|
||||||
|
return nil, func() {}, fmt.Errorf("file %s is missing chunks", file.Name)
|
||||||
|
}
|
||||||
|
n, copyErr := io.Copy(target, chunk)
|
||||||
|
closeErr := chunk.Close()
|
||||||
|
if copyErr != nil {
|
||||||
|
_ = target.Close()
|
||||||
|
cleanup()
|
||||||
|
return nil, func() {}, copyErr
|
||||||
|
}
|
||||||
|
if closeErr != nil {
|
||||||
|
_ = target.Close()
|
||||||
|
cleanup()
|
||||||
|
return nil, func() {}, closeErr
|
||||||
|
}
|
||||||
|
written += n
|
||||||
|
}
|
||||||
|
if err := target.Close(); err != nil {
|
||||||
|
cleanup()
|
||||||
|
return nil, func() {}, err
|
||||||
|
}
|
||||||
|
if written != file.Size {
|
||||||
|
cleanup()
|
||||||
|
return nil, func() {}, fmt.Errorf("assembled file size mismatch")
|
||||||
|
}
|
||||||
|
staged = append(staged, StagedUploadFile{
|
||||||
|
Filename: file.Name,
|
||||||
|
FileSize: file.Size,
|
||||||
|
MIMEType: file.ContentType,
|
||||||
|
Path: assembledPath,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return staged, cleanup, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resumableSessionWritable(session ResumableSession) error {
|
||||||
|
if session.Status != ResumableStatusUploading {
|
||||||
|
return fmt.Errorf("upload session is not active")
|
||||||
|
}
|
||||||
|
if !session.ExpiresAt.After(time.Now().UTC()) {
|
||||||
|
return fmt.Errorf("upload session expired")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func expectedChunkSize(fileSize, chunkSize int64, index int) int64 {
|
||||||
|
offset := int64(index) * chunkSize
|
||||||
|
remaining := fileSize - offset
|
||||||
|
if remaining < 0 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
if remaining > chunkSize {
|
||||||
|
return chunkSize
|
||||||
|
}
|
||||||
|
return remaining
|
||||||
|
}
|
||||||
|
|
||||||
|
func addChunkIndex(chunks []int, index int) []int {
|
||||||
|
for _, chunk := range chunks {
|
||||||
|
if chunk == index {
|
||||||
|
return chunks
|
||||||
|
}
|
||||||
|
}
|
||||||
|
chunks = append(chunks, index)
|
||||||
|
sort.Ints(chunks)
|
||||||
|
return chunks
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) resumableSessionDir(sessionID string) string {
|
||||||
|
return filepath.Join(s.dataDir, "tmp", "uploads", sessionID)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) resumableFileDir(sessionID, fileID string) string {
|
||||||
|
return filepath.Join(s.resumableSessionDir(sessionID), fileID)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) resumableChunkPath(sessionID, fileID string, index int) string {
|
||||||
|
return filepath.Join(s.resumableFileDir(sessionID, fileID), fmt.Sprintf("%06d.part", index))
|
||||||
|
}
|
||||||
@@ -42,6 +42,8 @@ type UploadOptions struct {
|
|||||||
ExpiresInMinutes int
|
ExpiresInMinutes int
|
||||||
MaxDownloads int
|
MaxDownloads int
|
||||||
Password string
|
Password string
|
||||||
|
PasswordSalt string
|
||||||
|
PasswordHash string
|
||||||
ObfuscateMetadata bool
|
ObfuscateMetadata bool
|
||||||
OwnerID string
|
OwnerID string
|
||||||
CollectionID string
|
CollectionID string
|
||||||
@@ -50,6 +52,56 @@ type UploadOptions struct {
|
|||||||
StorageBackendID string
|
StorageBackendID string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type IncomingFile interface {
|
||||||
|
Name() string
|
||||||
|
Size() int64
|
||||||
|
ContentType() string
|
||||||
|
Open() (io.ReadCloser, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type multipartIncomingFile struct {
|
||||||
|
header *multipart.FileHeader
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f multipartIncomingFile) Name() string {
|
||||||
|
return f.header.Filename
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f multipartIncomingFile) Size() int64 {
|
||||||
|
return f.header.Size
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f multipartIncomingFile) ContentType() string {
|
||||||
|
return f.header.Header.Get("Content-Type")
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f multipartIncomingFile) Open() (io.ReadCloser, error) {
|
||||||
|
return f.header.Open()
|
||||||
|
}
|
||||||
|
|
||||||
|
type StagedUploadFile struct {
|
||||||
|
Filename string
|
||||||
|
FileSize int64
|
||||||
|
MIMEType string
|
||||||
|
Path string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f StagedUploadFile) Name() string {
|
||||||
|
return f.Filename
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f StagedUploadFile) Size() int64 {
|
||||||
|
return f.FileSize
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f StagedUploadFile) ContentType() string {
|
||||||
|
return f.MIMEType
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f StagedUploadFile) Open() (io.ReadCloser, error) {
|
||||||
|
return os.Open(f.Path)
|
||||||
|
}
|
||||||
|
|
||||||
type Box struct {
|
type Box struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
OwnerID string `json:"ownerId,omitempty"`
|
OwnerID string `json:"ownerId,omitempty"`
|
||||||
@@ -198,6 +250,10 @@ func (s *UploadService) ValidateSize(size int64) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *UploadService) CreateBox(files []*multipart.FileHeader, opts UploadOptions) (UploadResult, error) {
|
func (s *UploadService) CreateBox(files []*multipart.FileHeader, opts UploadOptions) (UploadResult, error) {
|
||||||
|
return s.CreateBoxFromIncoming(multipartIncomingFiles(files), opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) CreateBoxFromIncoming(files []IncomingFile, opts UploadOptions) (UploadResult, error) {
|
||||||
if len(files) == 0 {
|
if len(files) == 0 {
|
||||||
return UploadResult{}, fmt.Errorf("no files were uploaded")
|
return UploadResult{}, fmt.Errorf("no files were uploaded")
|
||||||
}
|
}
|
||||||
@@ -232,13 +288,16 @@ func (s *UploadService) CreateBox(files []*multipart.FileHeader, opts UploadOpti
|
|||||||
}
|
}
|
||||||
deleteToken := randomID(32)
|
deleteToken := randomID(32)
|
||||||
box.DeleteTokenHash = deleteTokenHash(box.ID, deleteToken)
|
box.DeleteTokenHash = deleteTokenHash(box.ID, deleteToken)
|
||||||
if strings.TrimSpace(opts.Password) != "" {
|
if strings.TrimSpace(opts.PasswordHash) != "" {
|
||||||
|
box.PasswordSalt = opts.PasswordSalt
|
||||||
|
box.PasswordHash = opts.PasswordHash
|
||||||
|
} else if strings.TrimSpace(opts.Password) != "" {
|
||||||
salt, hash := hashPassword(opts.Password)
|
salt, hash := hashPassword(opts.Password)
|
||||||
box.PasswordSalt = salt
|
box.PasswordSalt = salt
|
||||||
box.PasswordHash = hash
|
box.PasswordHash = hash
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := s.writeFilesToBox(&box, files, opts); err != nil {
|
if err := s.writeIncomingFilesToBox(&box, files, opts); err != nil {
|
||||||
return UploadResult{}, err
|
return UploadResult{}, err
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -261,6 +320,10 @@ func (s *UploadService) CreateBox(files []*multipart.FileHeader, opts UploadOpti
|
|||||||
// selection into a single box). The box keeps its original expiry, password and
|
// selection into a single box). The box keeps its original expiry, password and
|
||||||
// other settings; only the new files are written.
|
// other settings; only the new files are written.
|
||||||
func (s *UploadService) AppendFiles(boxID string, files []*multipart.FileHeader, opts UploadOptions) (UploadResult, error) {
|
func (s *UploadService) AppendFiles(boxID string, files []*multipart.FileHeader, opts UploadOptions) (UploadResult, error) {
|
||||||
|
return s.AppendIncomingFiles(boxID, multipartIncomingFiles(files), opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) AppendIncomingFiles(boxID string, files []IncomingFile, opts UploadOptions) (UploadResult, error) {
|
||||||
if len(files) == 0 {
|
if len(files) == 0 {
|
||||||
return UploadResult{}, fmt.Errorf("no files were uploaded")
|
return UploadResult{}, fmt.Errorf("no files were uploaded")
|
||||||
}
|
}
|
||||||
@@ -268,7 +331,7 @@ func (s *UploadService) AppendFiles(boxID string, files []*multipart.FileHeader,
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return UploadResult{}, err
|
return UploadResult{}, err
|
||||||
}
|
}
|
||||||
if err := s.writeFilesToBox(&box, files, opts); err != nil {
|
if err := s.writeIncomingFilesToBox(&box, files, opts); err != nil {
|
||||||
return UploadResult{}, err
|
return UploadResult{}, err
|
||||||
}
|
}
|
||||||
if err := s.SaveBox(box); err != nil {
|
if err := s.SaveBox(box); err != nil {
|
||||||
@@ -289,14 +352,26 @@ func (s *UploadService) AppendFiles(boxID string, files []*multipart.FileHeader,
|
|||||||
// appends the file metadata to box.Files. The box's StorageBackendID determines
|
// appends the file metadata to box.Files. The box's StorageBackendID determines
|
||||||
// where files land, so it works for both new and existing boxes.
|
// where files land, so it works for both new and existing boxes.
|
||||||
func (s *UploadService) writeFilesToBox(box *Box, files []*multipart.FileHeader, opts UploadOptions) error {
|
func (s *UploadService) writeFilesToBox(box *Box, files []*multipart.FileHeader, opts UploadOptions) error {
|
||||||
|
return s.writeIncomingFilesToBox(box, multipartIncomingFiles(files), opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
func multipartIncomingFiles(files []*multipart.FileHeader) []IncomingFile {
|
||||||
|
incoming := make([]IncomingFile, 0, len(files))
|
||||||
|
for _, file := range files {
|
||||||
|
incoming = append(incoming, multipartIncomingFile{header: file})
|
||||||
|
}
|
||||||
|
return incoming
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *UploadService) writeIncomingFilesToBox(box *Box, files []IncomingFile, opts UploadOptions) error {
|
||||||
backend, err := s.storage.Backend(box.StorageBackendID)
|
backend, err := s.storage.Backend(box.StorageBackendID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, header := range files {
|
for _, incoming := range files {
|
||||||
if !opts.SkipSizeLimit {
|
if !opts.SkipSizeLimit {
|
||||||
if err := s.ValidateSize(header.Size); err != nil {
|
if err := s.ValidateSize(incoming.Size()); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -306,15 +381,15 @@ func (s *UploadService) writeFilesToBox(box *Box, files []*multipart.FileHeader,
|
|||||||
maxSize = 0
|
maxSize = 0
|
||||||
}
|
}
|
||||||
|
|
||||||
file, err := header.Open()
|
file, err := incoming.Open()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
fileID := randomID(8)
|
fileID := randomID(8)
|
||||||
storedName := "@each@" + fileID + strings.ToLower(filepath.Ext(header.Filename))
|
storedName := "@each@" + fileID + strings.ToLower(filepath.Ext(incoming.Name()))
|
||||||
objectKey := boxObjectKey(box.ID, storedName)
|
objectKey := boxObjectKey(box.ID, storedName)
|
||||||
contentType := header.Header.Get("Content-Type")
|
contentType := incoming.ContentType()
|
||||||
if contentType == "" {
|
if contentType == "" {
|
||||||
buffer := make([]byte, 512)
|
buffer := make([]byte, 512)
|
||||||
n, _ := file.Read(buffer)
|
n, _ := file.Read(buffer)
|
||||||
@@ -324,7 +399,7 @@ func (s *UploadService) writeFilesToBox(box *Box, files []*multipart.FileHeader,
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := s.writeUploadedObject(context.Background(), backend, objectKey, file, header.Size, maxSize, contentType); err != nil {
|
if err := s.writeUploadedObject(context.Background(), backend, objectKey, file, incoming.Size(), maxSize, contentType); err != nil {
|
||||||
file.Close()
|
file.Close()
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -332,9 +407,9 @@ func (s *UploadService) writeFilesToBox(box *Box, files []*multipart.FileHeader,
|
|||||||
|
|
||||||
box.Files = append(box.Files, File{
|
box.Files = append(box.Files, File{
|
||||||
ID: fileID,
|
ID: fileID,
|
||||||
Name: filepath.Base(header.Filename),
|
Name: filepath.Base(incoming.Name()),
|
||||||
StoredName: storedName,
|
StoredName: storedName,
|
||||||
Size: header.Size,
|
Size: incoming.Size(),
|
||||||
ContentType: contentType,
|
ContentType: contentType,
|
||||||
PreviewKind: previewKind(contentType),
|
PreviewKind: previewKind(contentType),
|
||||||
ObjectKey: objectKey,
|
ObjectKey: objectKey,
|
||||||
@@ -931,21 +1006,17 @@ func writeUploadedFile(path string, source multipart.File, maxSize int64) error
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *UploadService) writeUploadedObject(ctx context.Context, backend StorageBackend, key string, source multipart.File, size, maxSize int64, contentType string) error {
|
func (s *UploadService) writeUploadedObject(ctx context.Context, backend StorageBackend, key string, source io.Reader, size, maxSize int64, contentType string) error {
|
||||||
var reader io.Reader = source
|
var reader io.Reader = source
|
||||||
|
putSize := size
|
||||||
if maxSize > 0 {
|
if maxSize > 0 {
|
||||||
reader = io.LimitReader(source, maxSize+1)
|
if size > maxSize {
|
||||||
var buffer bytes.Buffer
|
|
||||||
written, err := io.Copy(&buffer, reader)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if written > maxSize {
|
|
||||||
return fmt.Errorf("file exceeds max upload size")
|
return fmt.Errorf("file exceeds max upload size")
|
||||||
}
|
}
|
||||||
return backend.Put(ctx, key, bytes.NewReader(buffer.Bytes()), written, contentType)
|
reader = io.LimitReader(source, maxSize)
|
||||||
|
putSize = size
|
||||||
}
|
}
|
||||||
return backend.Put(ctx, key, reader, size, contentType)
|
return backend.Put(ctx, key, reader, putSize, contentType)
|
||||||
}
|
}
|
||||||
|
|
||||||
func boxObjectKey(boxID, name string) string {
|
func boxObjectKey(boxID, name string) string {
|
||||||
|
|||||||
@@ -126,6 +126,166 @@ func TestLocalStorageBackendAndLegacyFallback(t *testing.T) {
|
|||||||
object.Body.Close()
|
object.Body.Close()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestResumableSessionUploadOutOfOrderAndComplete(t *testing.T) {
|
||||||
|
service := newTestUploadService(t)
|
||||||
|
session, err := service.CreateResumableSession([]ResumableFileInput{{
|
||||||
|
Name: "note.txt",
|
||||||
|
Size: 11,
|
||||||
|
ContentType: "text/plain",
|
||||||
|
Fingerprint: "sha256:first-chunk",
|
||||||
|
}}, UploadOptions{MaxDays: 1, Password: "secret"}, 4, time.Hour)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CreateResumableSession returned error: %v", err)
|
||||||
|
}
|
||||||
|
if session.Options.Password != "" || session.Options.PasswordHash == "" || session.Options.PasswordSalt == "" {
|
||||||
|
t.Fatalf("resumable session did not hash password before storage: %+v", session.Options)
|
||||||
|
}
|
||||||
|
if session.Files[0].ChunkCount != 3 {
|
||||||
|
t.Fatalf("ChunkCount = %d, want 3", session.Files[0].ChunkCount)
|
||||||
|
}
|
||||||
|
if session.Files[0].Fingerprint != "sha256:first-chunk" {
|
||||||
|
t.Fatalf("Fingerprint = %q", session.Files[0].Fingerprint)
|
||||||
|
}
|
||||||
|
for index, body := range map[int]string{2: "rld", 0: "hell", 1: "o wo"} {
|
||||||
|
updated, err := service.PutResumableChunk(testContext(), session.ID, session.Files[0].ID, index, strings.NewReader(body))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("PutResumableChunk(%d) returned error: %v", index, err)
|
||||||
|
}
|
||||||
|
if len(updated.Files[0].UploadedChunks) == 0 {
|
||||||
|
t.Fatalf("UploadedChunks was not updated")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result, completed, err := service.CompleteResumableSession(testContext(), session.ID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CompleteResumableSession returned error: %v", err)
|
||||||
|
}
|
||||||
|
if completed.Status != ResumableStatusCompleted || completed.BoxID != result.BoxID {
|
||||||
|
t.Fatalf("completed session = %+v, result = %+v", completed, result)
|
||||||
|
}
|
||||||
|
box := getTestBox(t, service, result.BoxID)
|
||||||
|
if box.PasswordHash == "" || box.PasswordSalt == "" || box.PasswordHash != session.Options.PasswordHash {
|
||||||
|
t.Fatalf("completed box did not preserve hashed password")
|
||||||
|
}
|
||||||
|
object, err := service.OpenFileObject(testContext(), box, box.Files[0])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("OpenFileObject returned error: %v", err)
|
||||||
|
}
|
||||||
|
data, err := io.ReadAll(object.Body)
|
||||||
|
object.Body.Close()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadAll returned error: %v", err)
|
||||||
|
}
|
||||||
|
if string(data) != "hello world" {
|
||||||
|
t.Fatalf("object body = %q", string(data))
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(service.resumableSessionDir(session.ID)); !os.IsNotExist(err) {
|
||||||
|
t.Fatalf("resumable temp dir after complete error = %v, want os.ErrNotExist", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResumableCompleteRejectsMissingChunks(t *testing.T) {
|
||||||
|
service := newTestUploadService(t)
|
||||||
|
session, err := service.CreateResumableSession([]ResumableFileInput{{
|
||||||
|
Name: "note.txt",
|
||||||
|
Size: 8,
|
||||||
|
ContentType: "text/plain",
|
||||||
|
}}, UploadOptions{MaxDays: 1}, 4, time.Hour)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CreateResumableSession returned error: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := service.PutResumableChunk(testContext(), session.ID, session.Files[0].ID, 0, strings.NewReader("hell")); err != nil {
|
||||||
|
t.Fatalf("PutResumableChunk returned error: %v", err)
|
||||||
|
}
|
||||||
|
if _, _, err := service.CompleteResumableSession(testContext(), session.ID); err == nil {
|
||||||
|
t.Fatalf("CompleteResumableSession accepted missing chunks")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResumableSessionCanAddFilesBeforeComplete(t *testing.T) {
|
||||||
|
service := newTestUploadService(t)
|
||||||
|
session, err := service.CreateResumableSession([]ResumableFileInput{{
|
||||||
|
Name: "one.txt",
|
||||||
|
Size: 4,
|
||||||
|
ContentType: "text/plain",
|
||||||
|
Fingerprint: "one",
|
||||||
|
}}, UploadOptions{MaxDays: 1}, 4, time.Hour)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CreateResumableSession returned error: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := service.PutResumableChunk(testContext(), session.ID, session.Files[0].ID, 0, strings.NewReader("one!")); err != nil {
|
||||||
|
t.Fatalf("PutResumableChunk one returned error: %v", err)
|
||||||
|
}
|
||||||
|
updated, err := service.AddResumableFiles(session.ID, []ResumableFileInput{{
|
||||||
|
Name: "two.txt",
|
||||||
|
Size: 4,
|
||||||
|
ContentType: "text/plain",
|
||||||
|
Fingerprint: "two",
|
||||||
|
}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("AddResumableFiles returned error: %v", err)
|
||||||
|
}
|
||||||
|
if len(updated.Files) != 2 {
|
||||||
|
t.Fatalf("files after add = %d, want 2", len(updated.Files))
|
||||||
|
}
|
||||||
|
if updated.Files[0].UploadedChunks[0] != 0 {
|
||||||
|
t.Fatalf("existing uploaded chunk was not preserved: %+v", updated.Files[0])
|
||||||
|
}
|
||||||
|
if _, err := service.AddResumableFiles(session.ID, []ResumableFileInput{{
|
||||||
|
Name: "two.txt",
|
||||||
|
Size: 4,
|
||||||
|
ContentType: "text/plain",
|
||||||
|
Fingerprint: "two",
|
||||||
|
}}); err != nil {
|
||||||
|
t.Fatalf("duplicate AddResumableFiles returned error: %v", err)
|
||||||
|
}
|
||||||
|
updated, err = service.GetResumableSession(session.ID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetResumableSession returned error: %v", err)
|
||||||
|
}
|
||||||
|
if len(updated.Files) != 2 {
|
||||||
|
t.Fatalf("duplicate add changed file count to %d", len(updated.Files))
|
||||||
|
}
|
||||||
|
if _, err := service.PutResumableChunk(testContext(), session.ID, updated.Files[1].ID, 0, strings.NewReader("two!")); err != nil {
|
||||||
|
t.Fatalf("PutResumableChunk two returned error: %v", err)
|
||||||
|
}
|
||||||
|
result, _, err := service.CompleteResumableSession(testContext(), session.ID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CompleteResumableSession returned error: %v", err)
|
||||||
|
}
|
||||||
|
box := getTestBox(t, service, result.BoxID)
|
||||||
|
if len(box.Files) != 2 {
|
||||||
|
t.Fatalf("completed box file count = %d, want 2", len(box.Files))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResumableCleanupRemovesExpiredSessionsAndChunks(t *testing.T) {
|
||||||
|
service := newTestUploadService(t)
|
||||||
|
session, err := service.CreateResumableSession([]ResumableFileInput{{
|
||||||
|
Name: "note.txt",
|
||||||
|
Size: 4,
|
||||||
|
ContentType: "text/plain",
|
||||||
|
}}, UploadOptions{MaxDays: 1}, 4, time.Millisecond)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CreateResumableSession returned error: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := service.PutResumableChunk(testContext(), session.ID, session.Files[0].ID, 0, strings.NewReader("hell")); err != nil {
|
||||||
|
t.Fatalf("PutResumableChunk returned error: %v", err)
|
||||||
|
}
|
||||||
|
cleaned, err := service.CleanupExpiredResumableSessions(time.Now().UTC().Add(time.Hour))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CleanupExpiredResumableSessions returned error: %v", err)
|
||||||
|
}
|
||||||
|
if cleaned != 1 {
|
||||||
|
t.Fatalf("cleaned = %d, want 1", cleaned)
|
||||||
|
}
|
||||||
|
if _, err := service.GetResumableSession(session.ID); !os.IsNotExist(err) {
|
||||||
|
t.Fatalf("GetResumableSession after cleanup error = %v, want os.ErrNotExist", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(service.resumableSessionDir(session.ID)); !os.IsNotExist(err) {
|
||||||
|
t.Fatalf("resumable temp dir after cleanup error = %v, want os.ErrNotExist", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestContaboStorageConfigAllowsDisplayNamesWithSpaces(t *testing.T) {
|
func TestContaboStorageConfigAllowsDisplayNamesWithSpaces(t *testing.T) {
|
||||||
service := newTestUploadService(t)
|
service := newTestUploadService(t)
|
||||||
cfg, err := service.Storage().CreateS3Backend(StorageBackendConfig{
|
cfg, err := service.Storage().CreateS3Backend(StorageBackendConfig{
|
||||||
|
|||||||
@@ -335,10 +335,13 @@ button {
|
|||||||
.file-progress-side {
|
.file-progress-side {
|
||||||
width: min(10rem, 32vw);
|
width: min(10rem, 32vw);
|
||||||
display: grid;
|
display: grid;
|
||||||
|
grid-template-columns: minmax(0, 1fr) auto;
|
||||||
gap: 0.35rem;
|
gap: 0.35rem;
|
||||||
|
align-items: center;
|
||||||
}
|
}
|
||||||
|
|
||||||
.file-progress-percent {
|
.file-progress-percent {
|
||||||
|
grid-column: 1 / -1;
|
||||||
color: var(--muted-foreground);
|
color: var(--muted-foreground);
|
||||||
font-size: 0.75rem;
|
font-size: 0.75rem;
|
||||||
text-align: right;
|
text-align: right;
|
||||||
@@ -349,6 +352,25 @@ button {
|
|||||||
margin-top: 0;
|
margin-top: 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.upload-file-remove {
|
||||||
|
width: 1.65rem;
|
||||||
|
height: 1.65rem;
|
||||||
|
min-height: 1.65rem;
|
||||||
|
padding: 0;
|
||||||
|
border-color: var(--border);
|
||||||
|
border-radius: 999px;
|
||||||
|
color: var(--muted-foreground);
|
||||||
|
background: var(--surface-1);
|
||||||
|
font-size: 1rem;
|
||||||
|
line-height: 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
.upload-file-remove:hover {
|
||||||
|
color: var(--foreground);
|
||||||
|
border-color: var(--primary);
|
||||||
|
background: var(--surface-1-hover);
|
||||||
|
}
|
||||||
|
|
||||||
.result-item small,
|
.result-item small,
|
||||||
.download-item small,
|
.download-item small,
|
||||||
.result-item code,
|
.result-item code,
|
||||||
|
|||||||
@@ -13,6 +13,7 @@
|
|||||||
const copyURL = document.querySelector("#copy-url");
|
const copyURL = document.querySelector("#copy-url");
|
||||||
const openBox = document.querySelector("#open-box");
|
const openBox = document.querySelector("#open-box");
|
||||||
const manageLink = document.querySelector("#manage-link");
|
const manageLink = document.querySelector("#manage-link");
|
||||||
|
const RESUMABLE_SESSIONS_KEY = "warpbox-resumable-sessions";
|
||||||
|
|
||||||
if (!form || !dropZone || !fileInput) {
|
if (!form || !dropZone || !fileInput) {
|
||||||
return;
|
return;
|
||||||
@@ -42,6 +43,7 @@
|
|||||||
|
|
||||||
let latestBoxURL = "";
|
let latestBoxURL = "";
|
||||||
let selectedFiles = [];
|
let selectedFiles = [];
|
||||||
|
let uploadLocked = false;
|
||||||
|
|
||||||
["dragenter", "dragover"].forEach((eventName) => {
|
["dragenter", "dragover"].forEach((eventName) => {
|
||||||
dropZone.addEventListener(eventName, (event) => {
|
dropZone.addEventListener(eventName, (event) => {
|
||||||
@@ -57,33 +59,52 @@
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
dropZone.addEventListener("drop", (event) => {
|
document.addEventListener("dragover", (event) => {
|
||||||
if (event.dataTransfer && event.dataTransfer.files.length > 0) {
|
if (event.dataTransfer && Array.from(event.dataTransfer.types || []).includes("Files")) {
|
||||||
fileInput.files = event.dataTransfer.files;
|
event.preventDefault();
|
||||||
updateSelectedState(event.dataTransfer.files);
|
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
fileInput.addEventListener("change", () => updateSelectedState(fileInput.files));
|
document.addEventListener("drop", (event) => {
|
||||||
|
if (!event.dataTransfer || !event.dataTransfer.files.length) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
event.preventDefault();
|
||||||
|
if (!dropZone.contains(event.target)) {
|
||||||
|
addSelectedFiles(event.dataTransfer.files);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
dropZone.addEventListener("drop", (event) => {
|
||||||
|
if (event.dataTransfer && event.dataTransfer.files.length > 0) {
|
||||||
|
addSelectedFiles(event.dataTransfer.files);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
fileInput.addEventListener("change", () => {
|
||||||
|
addSelectedFiles(fileInput.files);
|
||||||
|
fileInput.value = "";
|
||||||
|
});
|
||||||
|
|
||||||
form.addEventListener("submit", async (event) => {
|
form.addEventListener("submit", async (event) => {
|
||||||
event.preventDefault();
|
event.preventDefault();
|
||||||
if (!fileInput.files || fileInput.files.length === 0) {
|
if (selectedFiles.length === 0) {
|
||||||
updateStatus("Choose at least one file first.");
|
updateStatus("Choose at least one file first.");
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
const submit = form.querySelector("button[type='submit']");
|
const submit = form.querySelector("button[type='submit']");
|
||||||
const formData = new FormData(form);
|
const formData = uploadFormData();
|
||||||
selectedFiles = Array.from(fileInput.files);
|
|
||||||
renderQueue(selectedFiles, "queued");
|
renderQueue(selectedFiles, "queued");
|
||||||
setLoading(true, submit);
|
setLoading(true, submit);
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const payload = await uploadWithProgress(form.action, formData, selectedFiles);
|
const payload = await uploadResumable(form.action, formData, selectedFiles);
|
||||||
renderResult(payload);
|
renderResult(payload);
|
||||||
form.reset();
|
form.reset();
|
||||||
updateSelectedState([]);
|
selectedFiles = [];
|
||||||
|
fileInput.value = "";
|
||||||
|
updateSelectedState();
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
updateStatus(error.message || "Upload failed");
|
updateStatus(error.message || "Upload failed");
|
||||||
} finally {
|
} finally {
|
||||||
@@ -97,8 +118,27 @@
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
function updateSelectedState(files) {
|
function addSelectedFiles(files) {
|
||||||
selectedFiles = Array.from(files || []);
|
if (uploadLocked) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
Array.from(files || []).forEach((file) => {
|
||||||
|
if (!selectedFiles.some((existing) => fileIdentity(existing) === fileIdentity(file))) {
|
||||||
|
selectedFiles.push(file);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
updateSelectedState();
|
||||||
|
}
|
||||||
|
|
||||||
|
function removeSelectedFile(index) {
|
||||||
|
if (uploadLocked) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
selectedFiles.splice(index, 1);
|
||||||
|
updateSelectedState();
|
||||||
|
}
|
||||||
|
|
||||||
|
function updateSelectedState() {
|
||||||
const count = selectedFiles.length || 0;
|
const count = selectedFiles.length || 0;
|
||||||
const title = dropZone.querySelector(".drop-title");
|
const title = dropZone.querySelector(".drop-title");
|
||||||
if (title) {
|
if (title) {
|
||||||
@@ -116,6 +156,7 @@
|
|||||||
}
|
}
|
||||||
|
|
||||||
function setLoading(isLoading, submit) {
|
function setLoading(isLoading, submit) {
|
||||||
|
uploadLocked = isLoading;
|
||||||
if (progress) {
|
if (progress) {
|
||||||
progress.hidden = !isLoading;
|
progress.hidden = !isLoading;
|
||||||
}
|
}
|
||||||
@@ -201,18 +242,294 @@
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function uploadResumable(fallbackUrl, formData, files) {
|
||||||
|
if (!window.fetch || typeof Blob === "undefined") {
|
||||||
|
return uploadWithProgress(fallbackUrl, formData, files);
|
||||||
|
}
|
||||||
|
|
||||||
|
updateStatus("Fingerprinting files...");
|
||||||
|
const fingerprints = await Promise.all(files.map((file) => fileFingerprint(file)));
|
||||||
|
const createPayload = {
|
||||||
|
files: files.map((file, index) => ({
|
||||||
|
name: file.name,
|
||||||
|
size: file.size,
|
||||||
|
contentType: file.type || "application/octet-stream",
|
||||||
|
fingerprint: fingerprints[index],
|
||||||
|
})),
|
||||||
|
expiresMinutes: parseInt(formData.get("expires_minutes") || "0", 10) || 0,
|
||||||
|
maxDownloads: parseInt(formData.get("max_downloads") || "0", 10) || 0,
|
||||||
|
password: formData.get("password") || "",
|
||||||
|
obfuscateMetadata: formData.get("obfuscate_metadata") === "on",
|
||||||
|
collectionId: formData.get("collection_id") || "",
|
||||||
|
};
|
||||||
|
const persistable = !createPayload.password;
|
||||||
|
let session = persistable ? await findResumableSession(createPayload) : null;
|
||||||
|
if (session) {
|
||||||
|
session = await addMissingResumableFiles(session, createPayload);
|
||||||
|
}
|
||||||
|
if (!session || session.status !== "uploading") {
|
||||||
|
try {
|
||||||
|
session = await createResumableSession(createPayload);
|
||||||
|
} catch (error) {
|
||||||
|
if ((error.message || "").toLowerCase().includes("resumable uploads are disabled")) {
|
||||||
|
return uploadWithProgress(fallbackUrl, formData, files);
|
||||||
|
}
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
if (persistable) {
|
||||||
|
saveResumableSession(session, createPayload);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const sessionFiles = files.map((file, index) => matchSessionFile(session, createPayload.files[index]));
|
||||||
|
if (sessionFiles.some((file) => !file)) {
|
||||||
|
throw new Error("Upload session could not match the selected files");
|
||||||
|
}
|
||||||
|
|
||||||
|
updateStatus("Uploading...");
|
||||||
|
const totalBytes = files.reduce((sum, file) => sum + file.size, 0);
|
||||||
|
const completedByFile = new Array(files.length).fill(0);
|
||||||
|
sessionFiles.forEach((sessionFile, index) => {
|
||||||
|
completedByFile[index] = uploadedBytesForSessionFile(sessionFile, session.chunkSize);
|
||||||
|
setSingleFileProgress(index, files[index], percentForBytes(completedByFile[index], files[index].size));
|
||||||
|
});
|
||||||
|
setTotalProgress(percentForBytes(completedByFile.reduce((sum, bytes) => sum + bytes, 0), totalBytes));
|
||||||
|
|
||||||
|
for (let fileIndex = 0; fileIndex < files.length; fileIndex++) {
|
||||||
|
const file = files[fileIndex];
|
||||||
|
const sessionFile = sessionFiles[fileIndex];
|
||||||
|
const uploaded = new Set(sessionFile.uploadedChunks || []);
|
||||||
|
for (let chunkIndex = 0; chunkIndex < sessionFile.chunkCount; chunkIndex++) {
|
||||||
|
if (uploaded.has(chunkIndex)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const start = chunkIndex * session.chunkSize;
|
||||||
|
const end = Math.min(file.size, start + session.chunkSize);
|
||||||
|
await uploadChunk(session.sessionId, sessionFile.id, chunkIndex, file.slice(start, end), (loaded) => {
|
||||||
|
const currentTotal = completedByFile.reduce((sum, bytes) => sum + bytes, 0) + loaded;
|
||||||
|
setTotalProgress(percentForBytes(currentTotal, totalBytes));
|
||||||
|
setSingleFileProgress(fileIndex, file, percentForBytes(completedByFile[fileIndex] + loaded, file.size));
|
||||||
|
updateStatus(`${percentForBytes(currentTotal, totalBytes)}%`);
|
||||||
|
});
|
||||||
|
completedByFile[fileIndex] += end - start;
|
||||||
|
uploaded.add(chunkIndex);
|
||||||
|
sessionFile.uploadedChunks = Array.from(uploaded).sort((a, b) => a - b);
|
||||||
|
if (persistable) {
|
||||||
|
saveResumableSession(session, createPayload);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
setSingleFileProgress(fileIndex, file, 100);
|
||||||
|
}
|
||||||
|
|
||||||
|
const resultPayload = await completeResumableSession(session.sessionId);
|
||||||
|
if (persistable) {
|
||||||
|
removeResumableSession(session.sessionId);
|
||||||
|
}
|
||||||
|
setTotalProgress(100);
|
||||||
|
setFileProgress(files, 100);
|
||||||
|
return resultPayload;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function createResumableSession(payload) {
|
||||||
|
const response = await fetch("/api/v1/uploads/resumable", {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
"Accept": "application/json",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
},
|
||||||
|
body: JSON.stringify(payload),
|
||||||
|
});
|
||||||
|
return readUploadJSON(response, "Upload session could not be created");
|
||||||
|
}
|
||||||
|
|
||||||
|
async function fetchResumableStatus(sessionID) {
|
||||||
|
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}`, {
|
||||||
|
headers: { "Accept": "application/json" },
|
||||||
|
});
|
||||||
|
return readUploadJSON(response, "Upload session could not be resumed");
|
||||||
|
}
|
||||||
|
|
||||||
|
async function addResumableFiles(sessionID, files) {
|
||||||
|
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}/files`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: {
|
||||||
|
"Accept": "application/json",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
},
|
||||||
|
body: JSON.stringify({ files }),
|
||||||
|
});
|
||||||
|
return readUploadJSON(response, "Upload session files could not be added");
|
||||||
|
}
|
||||||
|
|
||||||
|
function uploadChunk(sessionID, fileID, chunkIndex, chunk, onProgress) {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const request = new XMLHttpRequest();
|
||||||
|
request.open("PUT", `/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}/files/${encodeURIComponent(fileID)}/chunks/${chunkIndex}`);
|
||||||
|
request.setRequestHeader("Accept", "application/json");
|
||||||
|
request.upload.addEventListener("progress", (event) => {
|
||||||
|
if (event.lengthComputable && onProgress) {
|
||||||
|
onProgress(event.loaded);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
request.addEventListener("load", () => {
|
||||||
|
if (request.status < 200 || request.status >= 300) {
|
||||||
|
let payload = {};
|
||||||
|
try {
|
||||||
|
payload = JSON.parse(request.responseText || "{}");
|
||||||
|
} catch (error) {
|
||||||
|
payload = {};
|
||||||
|
}
|
||||||
|
reject(new Error(payload.error || "Chunk upload failed"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
resolve();
|
||||||
|
});
|
||||||
|
request.addEventListener("error", () => reject(new Error("Network error during chunk upload")));
|
||||||
|
request.addEventListener("abort", () => reject(new Error("Chunk upload aborted")));
|
||||||
|
request.send(chunk);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async function completeResumableSession(sessionID) {
|
||||||
|
const response = await fetch(`/api/v1/uploads/resumable/${encodeURIComponent(sessionID)}/complete`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "Accept": "application/json" },
|
||||||
|
});
|
||||||
|
return readUploadJSON(response, "Upload could not be completed");
|
||||||
|
}
|
||||||
|
|
||||||
|
async function readUploadJSON(response, fallback) {
|
||||||
|
let payload = {};
|
||||||
|
try {
|
||||||
|
payload = await response.json();
|
||||||
|
} catch (error) {
|
||||||
|
payload = {};
|
||||||
|
}
|
||||||
|
if (!response.ok) {
|
||||||
|
throw new Error(payload.error || fallback);
|
||||||
|
}
|
||||||
|
return payload;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function findResumableSession(payload) {
|
||||||
|
const records = loadResumableSessions();
|
||||||
|
const optionKey = resumableOptionKey(payload);
|
||||||
|
const selectedKeys = new Set(payload.files.map((file) => resumableFileKey(file)));
|
||||||
|
for (const record of records) {
|
||||||
|
if (record.optionKey !== optionKey) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (!record.files || !record.files.some((file) => selectedKeys.has(resumableFileKey(file)))) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const session = await fetchResumableStatus(record.sessionId).catch(() => null);
|
||||||
|
if (!session || session.status !== "uploading") {
|
||||||
|
removeResumableSession(record.sessionId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const sessionKeys = new Set(session.files.map((file) => resumableFileKey(file)));
|
||||||
|
const sessionHasOnlySelectedFiles = session.files.every((file) => selectedKeys.has(resumableFileKey(file)));
|
||||||
|
const selectedContainsSessionFile = Array.from(sessionKeys).some((key) => selectedKeys.has(key));
|
||||||
|
if (sessionHasOnlySelectedFiles && selectedContainsSessionFile) {
|
||||||
|
return session;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function addMissingResumableFiles(session, payload) {
|
||||||
|
const existing = new Set(session.files.map((file) => resumableFileKey(file)));
|
||||||
|
const missing = payload.files.filter((file) => !existing.has(resumableFileKey(file)));
|
||||||
|
if (missing.length === 0) {
|
||||||
|
return session;
|
||||||
|
}
|
||||||
|
return addResumableFiles(session.sessionId, missing);
|
||||||
|
}
|
||||||
|
|
||||||
|
function matchSessionFile(session, file) {
|
||||||
|
const key = resumableFileKey(file);
|
||||||
|
return session.files.find((sessionFile) => resumableFileKey(sessionFile) === key) || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function resumableOptionKey(payload) {
|
||||||
|
return [
|
||||||
|
payload.expiresMinutes,
|
||||||
|
payload.maxDownloads,
|
||||||
|
payload.obfuscateMetadata ? "1" : "0",
|
||||||
|
payload.collectionId || "",
|
||||||
|
].join(":");
|
||||||
|
}
|
||||||
|
|
||||||
|
function resumableFileKey(file) {
|
||||||
|
return [file.name, file.size, file.fingerprint || ""].join(":");
|
||||||
|
}
|
||||||
|
|
||||||
|
function loadResumableSessions() {
|
||||||
|
try {
|
||||||
|
const value = localStorage.getItem(RESUMABLE_SESSIONS_KEY);
|
||||||
|
const records = value ? JSON.parse(value) : [];
|
||||||
|
return Array.isArray(records) ? records : [];
|
||||||
|
} catch (error) {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function saveResumableSession(session, payload) {
|
||||||
|
try {
|
||||||
|
const records = loadResumableSessions().filter((record) => record.sessionId !== session.sessionId);
|
||||||
|
records.push({
|
||||||
|
sessionId: session.sessionId,
|
||||||
|
optionKey: resumableOptionKey(payload),
|
||||||
|
files: session.files.map((file) => ({
|
||||||
|
name: file.name,
|
||||||
|
size: file.size,
|
||||||
|
fingerprint: file.fingerprint || "",
|
||||||
|
})),
|
||||||
|
updatedAt: new Date().toISOString(),
|
||||||
|
});
|
||||||
|
localStorage.setItem(RESUMABLE_SESSIONS_KEY, JSON.stringify(records.slice(-25)));
|
||||||
|
} catch (error) {
|
||||||
|
/* ignore persistence failures */
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function removeResumableSession(sessionID) {
|
||||||
|
try {
|
||||||
|
const records = loadResumableSessions().filter((record) => record.sessionId !== sessionID);
|
||||||
|
localStorage.setItem(RESUMABLE_SESSIONS_KEY, JSON.stringify(records));
|
||||||
|
} catch (error) {
|
||||||
|
/* ignore persistence failures */
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function uploadedBytesForSessionFile(file, chunkSize) {
|
||||||
|
return (file.uploadedChunks || []).reduce((sum, index) => {
|
||||||
|
const start = index * chunkSize;
|
||||||
|
const end = Math.min(file.size, start + chunkSize);
|
||||||
|
return sum + Math.max(0, end - start);
|
||||||
|
}, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
function percentForBytes(bytes, total) {
|
||||||
|
if (!total) {
|
||||||
|
return 100;
|
||||||
|
}
|
||||||
|
return Math.max(0, Math.min(100, Math.round((bytes / total) * 100)));
|
||||||
|
}
|
||||||
|
|
||||||
function renderQueue(files, status) {
|
function renderQueue(files, status) {
|
||||||
if (!uploadQueue) {
|
if (!uploadQueue) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
uploadQueue.hidden = files.length === 0;
|
uploadQueue.hidden = files.length === 0;
|
||||||
uploadQueue.replaceChildren();
|
uploadQueue.replaceChildren();
|
||||||
files.forEach((file) => {
|
files.forEach((file, index) => {
|
||||||
uploadQueue.append(createFileRow({
|
uploadQueue.append(createFileRow({
|
||||||
name: file.name,
|
name: file.name,
|
||||||
meta: window.Warpbox.formatBytes(file.size),
|
meta: window.Warpbox.formatBytes(file.size),
|
||||||
progress: status === "queued" ? 0 : 100,
|
progress: status === "queued" ? 0 : 100,
|
||||||
status,
|
status,
|
||||||
|
index,
|
||||||
|
removable: status === "queued",
|
||||||
}));
|
}));
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -221,6 +538,7 @@
|
|||||||
const row = document.createElement("div");
|
const row = document.createElement("div");
|
||||||
row.className = "result-item upload-file-row";
|
row.className = "result-item upload-file-row";
|
||||||
row.dataset.fileName = file.name;
|
row.dataset.fileName = file.name;
|
||||||
|
row.dataset.fileIndex = file.index || 0;
|
||||||
|
|
||||||
const body = document.createElement("span");
|
const body = document.createElement("span");
|
||||||
const name = document.createElement("strong");
|
const name = document.createElement("strong");
|
||||||
@@ -242,11 +560,47 @@
|
|||||||
fill.style.transform = `scaleX(${file.progress / 100})`;
|
fill.style.transform = `scaleX(${file.progress / 100})`;
|
||||||
bar.append(fill);
|
bar.append(fill);
|
||||||
side.append(percent, bar);
|
side.append(percent, bar);
|
||||||
|
if (file.removable) {
|
||||||
|
const remove = document.createElement("button");
|
||||||
|
remove.className = "upload-file-remove";
|
||||||
|
remove.type = "button";
|
||||||
|
remove.setAttribute("aria-label", `Remove ${file.name}`);
|
||||||
|
remove.textContent = "×";
|
||||||
|
remove.addEventListener("click", () => removeSelectedFile(file.index || 0));
|
||||||
|
side.append(remove);
|
||||||
|
}
|
||||||
|
|
||||||
row.append(body, side);
|
row.append(body, side);
|
||||||
return row;
|
return row;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function uploadFormData() {
|
||||||
|
const formData = new FormData(form);
|
||||||
|
formData.delete("file");
|
||||||
|
selectedFiles.forEach((file) => {
|
||||||
|
formData.append("file", file, file.name);
|
||||||
|
});
|
||||||
|
return formData;
|
||||||
|
}
|
||||||
|
|
||||||
|
function fileIdentity(file) {
|
||||||
|
return [file.name, file.size, file.lastModified || 0].join(":");
|
||||||
|
}
|
||||||
|
|
||||||
|
async function fileFingerprint(file) {
|
||||||
|
if (!window.crypto || !window.crypto.subtle || !file.slice || typeof TextEncoder === "undefined") {
|
||||||
|
return fileIdentity(file);
|
||||||
|
}
|
||||||
|
const sampleSize = Math.min(file.size, 1024 * 1024);
|
||||||
|
const sample = await file.slice(0, sampleSize).arrayBuffer();
|
||||||
|
const metadata = new TextEncoder().encode([file.name, file.size, file.lastModified || 0, sampleSize].join(":"));
|
||||||
|
const combined = new Uint8Array(metadata.byteLength + sample.byteLength);
|
||||||
|
combined.set(metadata, 0);
|
||||||
|
combined.set(new Uint8Array(sample), metadata.byteLength);
|
||||||
|
const digest = await window.crypto.subtle.digest("SHA-256", combined);
|
||||||
|
return Array.from(new Uint8Array(digest)).map((byte) => byte.toString(16).padStart(2, "0")).join("");
|
||||||
|
}
|
||||||
|
|
||||||
function setTotalProgress(percent) {
|
function setTotalProgress(percent) {
|
||||||
if (totalProgressBar) {
|
if (totalProgressBar) {
|
||||||
totalProgressBar.style.transform = `scaleX(${Math.max(0, Math.min(100, percent)) / 100})`;
|
totalProgressBar.style.transform = `scaleX(${Math.max(0, Math.min(100, percent)) / 100})`;
|
||||||
@@ -271,4 +625,23 @@
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function setSingleFileProgress(index, file, progress) {
|
||||||
|
if (!uploadQueue) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const row = uploadQueue.querySelector(`.upload-file-row[data-file-index="${index}"]`);
|
||||||
|
if (!row) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const percent = row.querySelector(".file-progress-percent");
|
||||||
|
const fill = row.querySelector(".file-progress span");
|
||||||
|
const normalized = Math.max(0, Math.min(100, progress));
|
||||||
|
if (percent) {
|
||||||
|
percent.textContent = `${normalized}%`;
|
||||||
|
}
|
||||||
|
if (fill) {
|
||||||
|
fill.style.transform = `scaleX(${normalized / 100})`;
|
||||||
|
}
|
||||||
|
}
|
||||||
})();
|
})();
|
||||||
|
|||||||
@@ -14,6 +14,10 @@
|
|||||||
<h2>Endpoints</h2>
|
<h2>Endpoints</h2>
|
||||||
<dl class="endpoint-list">
|
<dl class="endpoint-list">
|
||||||
<div><dt>Upload</dt><dd><code>POST /api/v1/upload</code></dd></div>
|
<div><dt>Upload</dt><dd><code>POST /api/v1/upload</code></dd></div>
|
||||||
|
<div><dt>Resumable create</dt><dd><code>POST /api/v1/uploads/resumable</code></dd></div>
|
||||||
|
<div><dt>Resumable status</dt><dd><code>GET /api/v1/uploads/resumable/{sessionID}</code></dd></div>
|
||||||
|
<div><dt>Resumable chunk</dt><dd><code>PUT /api/v1/uploads/resumable/{sessionID}/files/{fileID}/chunks/{index}</code></dd></div>
|
||||||
|
<div><dt>Resumable complete</dt><dd><code>POST /api/v1/uploads/resumable/{sessionID}/complete</code></dd></div>
|
||||||
<div><dt>Health</dt><dd><code>GET /health</code></dd></div>
|
<div><dt>Health</dt><dd><code>GET /health</code></dd></div>
|
||||||
<div><dt>Request schema</dt><dd><a href="/api/v1/schemas/upload-request.json"><code>/api/v1/schemas/upload-request.json</code></a></dd></div>
|
<div><dt>Request schema</dt><dd><a href="/api/v1/schemas/upload-request.json"><code>/api/v1/schemas/upload-request.json</code></a></dd></div>
|
||||||
<div><dt>Response schema</dt><dd><a href="/api/v1/schemas/upload-response.json"><code>/api/v1/schemas/upload-response.json</code></a></dd></div>
|
<div><dt>Response schema</dt><dd><a href="/api/v1/schemas/upload-response.json"><code>/api/v1/schemas/upload-response.json</code></a></dd></div>
|
||||||
@@ -21,6 +25,28 @@
|
|||||||
</div>
|
</div>
|
||||||
</article>
|
</article>
|
||||||
|
|
||||||
|
<article class="card docs-card docs-card-wide">
|
||||||
|
<div class="card-content">
|
||||||
|
<h2>Resumable uploads</h2>
|
||||||
|
<p>Browser uploads use the resumable API by default. Custom clients can use the same flow: create a session with file metadata, upload exact-sized chunks, then complete the session. Chunks are temporary and are cleaned if the session expires.</p>
|
||||||
|
<pre><code># 1. Create a session.
|
||||||
|
curl -s {{.Data.BaseURL}}/api/v1/uploads/resumable \
|
||||||
|
-H 'Accept: application/json' \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{"files":[{"name":"report.pdf","size":1048576,"contentType":"application/pdf"}],"expiresMinutes":1440}'
|
||||||
|
|
||||||
|
# 2. Upload each chunk using the returned sessionId, file id, and chunkSize.
|
||||||
|
dd if=./report.pdf bs=8388608 count=1 skip=0 2>/dev/null | \
|
||||||
|
curl -X PUT --data-binary @- \
|
||||||
|
{{.Data.BaseURL}}/api/v1/uploads/resumable/SESSION_ID/files/FILE_ID/chunks/0
|
||||||
|
|
||||||
|
# 3. Complete after all chunks are present. The response is the normal upload JSON.
|
||||||
|
curl -X POST -H 'Accept: application/json' \
|
||||||
|
{{.Data.BaseURL}}/api/v1/uploads/resumable/SESSION_ID/complete</code></pre>
|
||||||
|
<p class="muted-copy">For authenticated uploads, send the same <code>Authorization: Bearer <token></code> header on every resumable request. Incomplete chunks are stored under <code>data/tmp/uploads</code> before finalizing into the selected storage backend.</p>
|
||||||
|
</div>
|
||||||
|
</article>
|
||||||
|
|
||||||
<article class="card docs-card">
|
<article class="card docs-card">
|
||||||
<div class="card-content">
|
<div class="card-content">
|
||||||
<h2>Curl upload</h2>
|
<h2>Curl upload</h2>
|
||||||
|
|||||||
3
scripts/env/dev.env.example
vendored
3
scripts/env/dev.env.example
vendored
@@ -9,6 +9,9 @@ WARPBOX_CLEANUP_ENABLED=true
|
|||||||
WARPBOX_CLEANUP_EVERY=1h
|
WARPBOX_CLEANUP_EVERY=1h
|
||||||
WARPBOX_THUMBNAIL_ENABLED=true
|
WARPBOX_THUMBNAIL_ENABLED=true
|
||||||
WARPBOX_THUMBNAIL_EVERY=1m
|
WARPBOX_THUMBNAIL_EVERY=1m
|
||||||
|
WARPBOX_RESUMABLE_UPLOADS_ENABLED=true
|
||||||
|
WARPBOX_RESUMABLE_CHUNK_MB=16
|
||||||
|
WARPBOX_RESUMABLE_RETENTION_HOURS=1
|
||||||
WARPBOX_MAX_UPLOAD_SIZE_MB=16384
|
WARPBOX_MAX_UPLOAD_SIZE_MB=16384
|
||||||
WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true
|
WARPBOX_ANONYMOUS_UPLOADS_ENABLED=true
|
||||||
WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512
|
WARPBOX_ANONYMOUS_MAX_UPLOAD_MB=512
|
||||||
|
|||||||
Reference in New Issue
Block a user