warpbox-dev/docs/Design Docs/Warpbox.dev Design & Requirements – Self‑Hosted File Transfer Platform(1).md

# Warpbox.dev Design & Requirements – Self‑Hosted File Transfer Platform

## 1. Vision and Goals

Warpbox.dev is a self‑hosted, open‑source file transfer and lightweight file hosting service inspired by PsiTransfer, transfer.sh, and Gofile, but built entirely in Go with a minimal vanilla JS/CSS frontend. It must be easy to rebrand and run as a multi‑tenant or single‑tenant instance (e.g., `warpbox.dev`) while also supporting fully self‑hosted private deployments via Docker.[^1][^2][^3]

Core goals:

- Anonymous, frictionless uploads (similar to PsiTransfer / transfer.sh).[^2][^3][^1]
- Optional “account mode” with folders, file management, and dashboards similar to Gofile.[^4][^5][^6]
- Admin console for moderation, statistics, and storage management.[^7]
- First‑class CLI and curl upload support for automation.[^8][^9][^2]
- High‑quality link previews (Discord, social, chat apps) with working image/video embeds where possible via Open Graph/Twitter Card metadata.[^10][^11][^12]
- Simple, secure API for custom clients, including ShareX integration.


## 2. High‑Level Feature Set

### 2.1 Anonymous Upload Mode

Anonymous upload mode mimics PsiTransfer and transfer.sh: anyone can upload without an account and immediately receive a shareable link.[^3][^1][^2]

Required features:

- Drag‑and‑drop web UI for uploading one or many files.
- Upload very large files (configurable max size; resumable where possible).
- Generated “bucket”/“upload” ID with random token in URL (e.g., `/d/{id}` or `/b/{id}`).
- Optional one‑time download links and max downloads / max days, similar to transfer.sh/PsiTransfer.[^13][^2][^3]
- Automatic expiration and cleanup based on configured policies.
- Optional password‑protected buckets (AES or equivalent; password known only to the user, stored hashed/encrypted server‑side), like PsiTransfer’s password‑protected download list.[^14][^13][^3]
- Option to download all files in a bucket as zip/tar.gz.[^13][^3]

Nice‑to‑have:

- Simple frontpage with “quick upload”, recent uploads for the current browser session only, and copy‑link buttons.
- Human‑readable short URLs in addition to long random IDs.

### 2.2 Authenticated User Mode ("Self‑Hosted Gofile")

Authenticated mode should approximate a lightweight, self‑hosted Gofile: users can register/log in, create folders, and manage uploads.[^5][^6][^15][^4]

Required features:

- User accounts (email + password, or OAuth providers as optional later).
- Folder hierarchy for each user (root + nested folders).
- Upload files into specific folders from the web UI.
- Per‑file metadata: filename, size, content‑type, upload date, last accessed, owner, visibility (public/unlisted/private), expiration, password.
- Per‑folder metadata: name, created at, parent folder, total size, number of files.
- File actions: rename, move, delete, set expiration, set visibility, set password.
- Folder actions: rename, move, delete (with safety checks), share folder as “public gallery”.
- User dashboard: list of uploads, sort/filter, basic stats (total storage, bandwidth used, recent activity).

Nice‑to‑have:

- Guest tokens like Gofile uses for pseudo‑accounts attached to a browser session.[^16]
- Tagging of files and folders.
- Simple search (by filename, extension, MIME type, tags) within a user’s space.

### 2.3 Admin Backend & Management

Admin capabilities should go beyond PsiTransfer’s simple `/admin` bucket list and provide real operational visibility.[^1][^7][^13]

Required features:

- Admin authentication (separate role from regular users; ideally role‑based access control with roles such as `admin`, `moderator`, `support`).
- Admin dashboard with:
  - Total files, total size on disk, total bandwidth used.
  - Daily/weekly/monthly uploads and downloads.
  - Top files by size, downloads, or bandwidth.
  - Top users by storage usage and bandwidth.
- Bucket/file management:
  - Search by ID, filename, user, IP, or date range.
  - View metadata and status of an upload bucket.
  - Force delete file/bucket.
  - Extend or shorten expiration date.
  - Lock or disable a user account.
- Abuse/moderation tools:
  - Flag content as suspected TOS violation.
  - Mark user or IP as blocked (no further uploads).
  - Optionally integrate with external abuse reporting or virus scanning services (later).

Nice‑to‑have:

- Audit logs of admin actions.
- Config UI for system‑wide settings: max file size, default expiration, storage backend, mail settings, etc.


## 3. Upload and Download Flows

### 3.1 Web Uploads

Web uploads should behave similarly to PsiTransfer: responsive UI, multiple files, resumable uploads where possible.[^14][^3][^13]

Core behaviors:

- Drag‑and‑drop multiple files or folder selection.
- Progress bars per file and aggregated progress.
- Resumable uploads for large files using tus.io or a similar resumable upload protocol, which is widely used in PsiTransfer‑like tools and other self‑hosted solutions.[^17][^14]
- Graceful error handling: network failures, timeouts, file too large, storage quota exceeded.

### 3.2 CLI / Curl Uploads

Curl upload support is a hard requirement and should be as simple as transfer.sh’s interface.[^18][^9][^8][^2]

Required behaviors:

- Basic upload:
  
  ```bash
  curl --upload-file ./file.ext https://warpbox.dev/file.ext
  ```

  Returns a direct URL on stdout (e.g., `https://warpbox.dev/d/{id}/file.ext`).[^19][^2]

- Multiple files via multipart form:
  
  ```bash
  curl -F file=@file1 -F file=@file2 https://warpbox.dev/upload
  ```

- HTTP headers or query parameters for options, similar to transfer.sh:
  
  - `Max-Downloads` – max download count before expiration.
  - `Max-Days` – max lifetime in days.[^2]
  - Optional `X-Warpbox-Password` – to set a download password.
  - Optional `X-Warpbox-Folder` – target folder for authenticated uploads.

- Simple one‑liner helper function in docs (bash/zsh/fish/PowerShell) inspired by transfer.sh examples.[^9][^2]

Nice‑to‑have:

- `warpbox` CLI wrapper (Go binary) as an alternative to raw curl.

### 3.3 Download Behavior

Downloads must be simple but efficient and support range requests for partial downloads/resume.[^14]

Required behaviors:

- Support HTTP range requests (`Accept-Ranges`) so download managers and browsers can resume incomplete downloads.[^14]
- Correct `Content-Type`, `Content-Length`, and `Content-Disposition` headers.
- Optional streaming mode for media (see section 5) when content type is `video/*` or `image/*`.
- Respect max‑downloads and expiration on every request.


## 4. API Design

### 4.1 General Principles

The API should be a simple JSON‑over‑HTTP REST API secured with token‑based authentication for privileged operations.

General conventions:

- Base path: `/api/v1`.
- Authentication: 
  - User API tokens (per account) for user endpoints.
  - Admin API tokens (different scope) for admin endpoints.
- Request/response: JSON bodies for metadata, multipart/form‑data for file upload.
- Pagination for list endpoints (`page`, `pageSize` or `limit`/`offset`).

### 4.2 Public/Anonymous Endpoints

- `POST /api/v1/upload`
  - Anonymous upload.
  - Accepts multipart form `file` field(s).
  - Optional form fields or headers: `max_downloads`, `max_days`, `password`, `filename`.
  - Response: bucket ID, list of file URLs, expiration.

- `GET /api/v1/buckets/{id}`
  - Returns limited metadata about anonymous bucket if public.

- `DELETE /api/v1/buckets/{id}` (with delete token)
  - If bucket was created anonymously, return a management/delete token in the response URL (similar to Gofile management links and other anonymous file hosts).[^4]

### 4.3 Authenticated User Endpoints

- `POST /api/v1/auth/login`
- `POST /api/v1/auth/register`
- `POST /api/v1/auth/refresh`
- `GET /api/v1/me` – profile, usage quotas.

Folders:

- `GET /api/v1/folders` – list root folders.
- `POST /api/v1/folders` – create folder.
- `GET /api/v1/folders/{id}` – folder details + paginated files.
- `PATCH /api/v1/folders/{id}` – rename/move.
- `DELETE /api/v1/folders/{id}`.

Files:

- `POST /api/v1/folders/{id}/upload` – upload into folder via multipart.
- `GET /api/v1/files/{id}` – file metadata.
- `PATCH /api/v1/files/{id}` – change filename, visibility, expiration, password, folder.
- `DELETE /api/v1/files/{id}`.

### 4.4 Admin Endpoints

- `GET /api/v1/admin/stats/overview`
  - Global metrics: total size, total files, uploads/downloads per day.[^7]

- `GET /api/v1/admin/files` / `GET /api/v1/admin/buckets`
  - Search/filter by user, size, date, IP, status.

- `PATCH /api/v1/admin/files/{id}`
  - Force delete, change expiration, lock.

- `GET /api/v1/admin/users`
- `PATCH /api/v1/admin/users/{id}` – change roles, lock user.

- `GET /api/v1/admin/logs` – optional; audit logs.


## 5. Embeds and Social Link Previews

### 5.1 Discord and Social Embeds Basics

Discord and other platforms primarily rely on Open Graph meta tags (`og:*`) and Twitter Card tags for rich embeds.[^11][^12][^20][^10]

Key tags supported by Discord:

- `og:title` – title of the page.
- `og:description` – description text under the title.
- `og:image` – preview image.
- `og:url` – canonical URL.
- `og:site_name` – small text above the title.
- Optional `theme-color` – sets the colored left border of the embed in Discord.[^12][^11]

Discord also reads Twitter Card tags, with `twitter:card` recommended as `summary_large_image` for large previews.[^10][^12]

### 5.2 Behavior for Image and Video Files

Image embeds:

- For `image/*` files, serve a landing page (`/v/{id}`) that includes:
  - `og:type = "website"`.
  - `og:title` = filename or user‑supplied title.
  - `og:description` = short description or “Shared via Warpbox”.
  - `og:image` = direct URL of the image.
  - `og:url` = landing page URL.
  - `og:site_name` = brand/site name.
  - `theme-color` = configurable brand color.[^11][^12][^10]

Video embeds:

- Many platforms restrict video embeds to a whitelist of providers; Discord only allows video playback via OG/OEmbed for a small set of providers like YouTube.[^21][^20][^12]
- For generic video files (`video/*`), fully inline video playback in Discord is not reliable, but the best practice is:
  - Set `og:image` to a generated thumbnail.
  - Use `og:title`, `og:description`, `og:url` as above.
  - Provide a direct download/stream link in the embed description.

Document and generic files:

- Use a generic icon preview or first‑page thumbnail (for PDFs) as `og:image`.

### 5.3 OEmbed Considerations

Discord supports oEmbed but only for whitelisted providers; for most self‑hosted sites, Open Graph + Twitter Card tags are the primary mechanism.[^20][^12][^11]

Nice‑to‑have:

- Provide an OEmbed endpoint (`/oembed`) to maximize compatibility with clients that do support custom providers.


## 6. ShareX and Desktop Integration

### 6.1 ShareX Custom Uploader

ShareX can integrate with custom uploaders via `.sxcu` configuration files, which specify the request URL, file field name, headers, and URL patterns.[^22][^23][^24][^25]

Requirements:

- HTTP endpoint suitable for ShareX:
  - `POST https://warpbox.dev/api/v1/upload` for anonymous.
  - `POST https://warpbox.dev/api/v1/folders/{id}/upload` for authenticated.
- Accept multipart form field `file` or `sharex` (configurable) with the uploaded file.[^25]
- Support `Authorization` header with API key/token, as many setups do.[^24][^22][^25]
- Respond with JSON containing a direct link, e.g. `{ "url": "https://warpbox.dev/d/{id}/file.ext" }`.

The project should ship example `.sxcu` files and documentation on how to import them, matching typical ShareX guides.[^23][^22][^25]

### 6.2 Warpbox.exe Helper

Warpbox should ship a small `warpbox.exe` for Windows and an equivalent shell script for Linux/macOS.

Concept:

- `warpbox.exe` is a simple CLI wrapper around curl that:
  - Reads configuration (server URL, API key, default folder) from a config file.
  - Provides basic commands: `warpbox upload file.ext`, `warpbox url` etc.
  - Optionally generates a `.sxcu` file tailored to the instance.

- For Linux/macOS, a `warpbox.sh` shell script:
  - Validates presence of `curl`.
  - Wraps example curl commands.

- Documentation:
  - Step‑by‑step instructions to set up ShareX with Warpbox using the `.sxcu` file and API token, based on common patterns in existing tutorials.[^22][^23][^24][^25]


## 7. Architecture Overview (Golang + Vanilla JS/CSS)

### 7.1 Backend Architecture (Go)

Core stack:

- Language: Go.
- HTTP framework: standard library `net/http` or lightweight router (e.g., chi, gorilla/mux) as an implementation detail.
- Persistence options:
  - PostgreSQL or MySQL for structured data (users, files, folders, tokens, stats).
  - Optional SQLite for single‑user or small instances.
- Storage backend:
  - Local filesystem (mount volume in Docker) for simple setups.
  - Pluggable S3‑compatible backend for scalable deployments, similar to how transfer.sh supports S3.[^26]

Major components:

- API server: handles REST endpoints, authentication, and business logic.
- Upload engine: streaming uploads to disk/S3; optional resumable uploads (tus.io‑like protocol implementation or existing Go tus library).[^17][^14]
- Download server: efficient range‑enabled file serving.
- Admin module: stats aggregation, moderation, storage quotas.
- Background workers:
  - Expiration/cleanup of old files and buckets.
  - Pre‑generation of thumbnails for images/videos.
  - Periodic stats aggregation for dashboard.

### 7.2 Frontend Architecture (Vanilla JS/CSS)

The frontend should be framework‑free to minimize footprint and keep rebranding simple.

Key pages:

- Landing / anonymous upload.
- Auth pages: login, register, password reset.
- User dashboard (folders/files grid or list view).
- File/bucket details view.
- Admin dashboard.

Implementation notes:

- Use vanilla JS `fetch` API for calling the backend.
- Minimal build tooling; optionally a simple bundler for minification.
- CSS:
  - Modular BEM‑style classes, or simple utility classes.
  - Dark/light themes with CSS variables for easy rebranding.


## 8. Deployment, Docker, and From‑Scratch Image

### 8.1 Docker Image Strategy

Warpbox should ship an official Docker image with a multi‑stage build, which is common best practice for Go services.[^26]

Recommended Dockerfile approach:

1. **Builder stage**
   - Base: `golang:1.x-alpine`.
   - Fetch dependencies, build statically linked binary with `CGO_ENABLED=0`.

2. **Runner stage**
   - Base: `FROM scratch`.
   - Copy binary from builder.
   - Copy static assets (HTML, JS, CSS) into an internal directory.
   - Expose port (e.g., 8080).
   - Set entrypoint to the Go binary.

This keeps the runtime image extremely small while supporting a fully self‑contained binary.[^26]

### 8.2 Configuration and Environment

- Configuration via environment variables:
  - `WARPBOX_DB_DSN`, `WARPBOX_STORAGE_PATH` or S3 credentials.
  - `WARPBOX_MAX_FILE_SIZE`, `WARPBOX_DEFAULT_EXPIRATION_DAYS`.
  - `WARPBOX_BRAND_NAME`, `WARPBOX_BRAND_COLOR` for rebranding.
  - `WARPBOX_ADMIN_EMAIL`, `WARPBOX_ADMIN_PASSWORD` for bootstrapping.

- Provide Docker Compose examples for:
  - Warpbox + Postgres + Traefik/NGINX reverse proxy.
  - Warpbox with S3 backend.


## 9. Security, Privacy, and Abuse Handling

### 9.1 Security Baselines

- All access via HTTPS (left to reverse proxy, but documented).
- Strong password hashing (Argon2id, bcrypt) for user credentials.
- CSRF protection for web UI forms (if cookies used).
- Strict API authentication with bearer tokens over HTTPS.
- Rate limiting for anonymous uploads to prevent abuse.

Nice‑to‑have:

- Content‑type and file extension validation.
- Optional integration with virus scanning (e.g., ClamAV or third‑party services), similar to transfer.sh’s optional virus scanning feature.[^26]

### 9.2 Privacy and Anonymity

- Anonymous uploads should not require registration and may log minimal metadata (IP, user agent) only for abuse prevention.
- Clear configuration for retention of logs and metadata.
- Optional “privacy mode” where IPs are hashed or truncated.

### 9.3 Abuse and Legal Considerations

- Admin tools to quickly remove illegal or infringing content and ban abusive users/IPs.[^7]
- TOS/Acceptable Use Policy recommended for public instances.


## 10. SEO and Discoverability

Even though Warpbox links are primarily shared directly, basic SEO ensures public pages index correctly.

Required SEO features:

- Proper `<title>` and `<meta name="description">` tags on public/view pages.
- Canonical URLs using `>`.
- Open Graph and Twitter Card tags as described in section 5.[^12][^10]
- `robots.txt` with sensible defaults (e.g., disallow admin endpoints, allow public galleries).
- Sitemap for discoverable public galleries (optional, enabled only for public instances).


## 11. Rebranding and Theming

Warpbox should be easy to fork and rebrand, or to configure branding at runtime.

Required:

- All exposed names, titles, and meta tags use configurable brand name.
- Primary colors defined as CSS variables and customizable via config.
- Logo and favicon paths configurable (e.g., mount custom assets in Docker volume).

Nice‑to‑have:

- Simple theme system (JSON/YAML theme configs) to quickly switch branding.


## 12. Quality‑of‑Life Features and Future Enhancements

Potential QoL features inspired by existing self‑hosted file sharing platforms and user expectations:[^13][^1][^7][^26]

- Drag‑and‑drop upload that works across desktop and mobile.
- Clipboard helpers: one‑click copy link, copy embed code.
- QR code generation for download URLs.
- Simple “text paste” mode for sharing snippets (Gist‑like).
- Optional email notification to the uploader when a file is first downloaded.
- Light‑weight analytics for individual files (download count, last download time).
- Multi‑language support (i18n).
- Public folder galleries with grid view of images/videos.

These are not necessary for v1 but can significantly improve the perceived polish and usability of Warpbox.


## 13. Minimal Viable Product (MVP) Scope

To keep the first version shippable, the following is a suggested MVP:

- Anonymous file uploads (web + curl) with:
  - Max file size config.
  - Expiration (by days) and optional max downloads.
  - Basic landing page with Open Graph image preview.
- Simple user accounts with:
  - Single‑level “folders” or collections.
  - Upload into folders via web UI.
  - Basic dashboard listing files and folders.
- Admin panel:
  - Login.
  - Global stats (total size, total files, recent uploads).
  - Search and delete files.
- API:
  - `POST /api/v1/upload` for anonymous.
  - Basic authenticated upload and file management endpoints.
- Docker multi‑stage build with `FROM scratch` runner and minimal configuration variables.

Future iterations can then add:

- Resumable uploads.
- Advanced admin analytics.
- ShareX presets and warpbox.exe.
- S3 backend and multi‑node scaling.
- Thumbnails, galleries, and richer embeds.

---

## References

1. [GitHub - psi-4ward/psitransfer: Simple open source self-hosted file ...](https://github.com/psi-4ward/psitransfer) - Simple open source self-hosted file sharing solution. It's an alternative to paid services like Drop...

2. [transfer.sh](https://transfer.sh) - Easy and fast file sharing from the command-line.

3. [PSiTransfer - A Simple Open Source Self-hosted File Sharing Solution](https://ostechnix.com/psitransfer-simple-open-source-self-hosted-file-sharing-solution/) - PSiTransfer is a simple and open source file sharing utility used to share your files locally or glo...

4. [Gofile.io 匿名文件分享平台，不限文件大小和下载速度- 网盘](https://uzbox.com/en/www/upload/gofile-html) - Gofile Introduction Gofile.io is a free and anonymous file sharing platform where users can quickly ...

5. [Understanding Gofile: A Gateway to Anonymous File Sharing - Oreate AIwww.oreateai.com › blog › understanding-gofile-a-gateway-to-anonymous...](https://www.oreateai.com/blog/understanding-gofile-a-gateway-to-anonymous-file-sharing/9c83fd65ea686b69aa88ebe3ced1944b) - Gofile.io offers free anonymous file sharing with no data limits while emphasizing privacy through e...

6. [Gofile.io: Free, anonymous file sharing with no limits - AlternativeTo](https://alternativeto.net/software/gofile-io/about/) - Enjoy free, anonymous file sharing without data limits. Upload, store, and stream media files at max...

7. [Erugo - Self-Hosted File Sharing Platform](https://erugo.app) - Erugo is a powerful, self-hosted file-sharing platform that puts you in complete control of your dat...

8. [transfer.sh command - github.com/dutchcoders ...](https://pkg.go.dev/github.com/dutchcoders/transfer.sh)

9. [transfer.sh/examples.md at main · dutchcoders/transfer.sh](https://github.com/dutchcoders/transfer.sh/blob/main/examples.md) - Easy and fast file sharing from the command-line. Contribute to dutchcoders/transfer.sh development ...

10. [How to Add Social Media Embeds To Your Website (Open Graph ...](https://www.howtogeek.com/devops/how-to-add-social-media-embeds-to-your-website-open-graph-meta-tags/) - The configuration for these embeds is done using <meta> tags, usually in the header of your site. Th...

11. [Discord Website Embeds - Nora's Hideout - GitHub Pages](https://analogfeelings.github.io/knowledge/discord-website-embeds/) - A Discord website embed is composed of several HTML meta tags, and optionally, an OEmbed JSON file. ...

12. [Discord Embed Meta Tags: Complete Reference - opengraphplus.com](https://opengraphplus.com/consumers/discord/tags) - Discord supports oEmbed, but only for whitelisted providers (YouTube, Twitch, Spotify, etc.). For mo...

13. [PsiTransfer is a Free Self-hosted File Transfer Solution - MEDevel.com](https://medevel.com/psitransfer-x/) - Features. No accounts, no logins; Mobile friendly responsive interface; Supports many and very big f...

14. [PsiTransfer - psi.cx](https://psi.cx/2017/psitransfer/) - Simple open source self-hosted file sharing solution with robust, resumeable up- and downloads of la...

15. [Send Large Files - Up to 5GB Free - Gofile](https://gofile.to/about) - Gofile is the simplest way to send your files around the world. We host the Internet!

16. [KasimKaizer/gofileioupload: A simple package to upload ...](https://github.com/KasimKaizer/gofileioupload) - A simple package to upload files to gofile.io. Contribute to KasimKaizer/gofileioupload development ...

17. [Self hosted file transfer that can resume from connection interruptions.](https://www.reddit.com/r/selfhosted/comments/1qhb5l5/self_hosted_file_transfer_that_can_resume_from/) - Hi, I'm new to this subreddit and hope this is the right place to ask this. I am trying to host my o...

18. [Upload files from command line using tranfer.sh](https://blog.kiprosh.com/upload-files-from-command-line-using-tranfer-sh/) - Ever stuck on a remote server with some file? Needed to upload/download file from remote server? Fin...

19. [GitHub - jahands/transfer.sh: Easy and fast file sharing from the command-line.](https://github.com/jahands/transfer.sh) - Easy and fast file sharing from the command-line. Contribute to jahands/transfer.sh development by c...

20. [What standard does Discord use when generating embeds from links?](https://www.reddit.com/r/discordapp/comments/1hd7mku/what_standard_does_discord_use_when_generating/) - What meta-tags (I suppose it's meta tags, anyway) does Discord look for when generating things like ...

21. [How can I add an OG meta tag that embeds a video? - Stack Overflow](https://stackoverflow.com/questions/66478860/how-can-i-add-an-og-meta-tag-that-embeds-a-video) - I have been trying to embed a video in my HTML page using OG Meta Tag and it does not work apparentl...

22. [ShareX | S.EE Docs](https://s.ee/docs/sharex/) - Open ShareX · Go to Destinations > Custom uploader settings · Click Import, then either: Select the ...

23. [Setting up ShareX and Image Uploaders | New Day RP](https://newdayrp.com/threads/setting-up-sharex-and-image-uploaders.66990/) - Here are a few general guides to using ShareX with an uploader service, along with general recommend...

24. [Custom domain for your images with ShareX - DEV Community](https://dev.to/thomasbnt/custom-domain-for-your-images-with-sharex-3bmi) - To upload your images, you need to make it request on your server and especially on a file. There ar...

25. [ShareX CUSTOM Domain/URL Setup - YouTube](https://www.youtube.com/watch?v=CeXy3iB-aHA) - How to setup ShareX with your own custom domain or URL, this guide details the configuration and req...

26. [transfer.sh Deploy Guide - Zeabur](https://zeabur.com/templates/OMT6RO) - Key Features · Upload files from terminal with curl, wget, or any HTTP client · Built-in web upload ...