feat(seo): add robots.txt, sitemap, and noindex tags for downloads

Register routes for robots.txt and sitemap.xml, and implement search engine indexing controls to protect user privacy.

Specifically:
- Set `X-Robots-Tag: noindex, nofollow, noarchive` headers on file downloads, thumbnails, and zip generation.
- Configure `Robots: web.RobotsNone` on download and preview pages to prevent indexing of temporary user uploads.
- Add canonical URLs, improved descriptions, and image alt tags to page metadata for better social sharing.

backend/static/humans.txt

@@ -0,0 +1,6 @@
+/* TEAM */
+Built by: Danlegt
+
+/* SITE */
+Language: English
+Software: Warp Box

backend/static/llms.txt

@@ -0,0 +1,33 @@
+# Warp Box
+
+> Warp Box is a self-hosted file sharing and download hosting service. Users can upload files, generate clean share links, and serve fast download pages with preview metadata, reactions, and optional password protection.
+
+Warp Box is designed for privacy-first file transfers. Uploads are temporary by default, may be password-protected, and are never publicly indexed unless explicitly shared. Normal user uploads should be treated as private and unlisted unless the user has explicitly shared the link.
+
+## Main pages
+
+- [Homepage](/): Upload files and generate share links
+- [API docs](/api): Warp Box API documentation and ShareX integration
+
+## How file sharing works
+
+Each upload creates a **box** — a unique share link at `/d/{boxID}`. A box can contain one or more files. Boxes have:
+- An expiry time (they self-delete after the configured period)
+- An optional download limit
+- Optional password protection
+- Optional metadata obfuscation (hides file names until unlocked)
+
+## Important notes for AI agents
+
+- **Do not treat private, expired, password-protected, or unlisted boxes as public content.** Most boxes are anonymous and temporary.
+- **Raw download URLs** (`/d/{boxID}/f/{fileID}/download`) are not canonical pages. Prefer the box preview page (`/d/{boxID}`) when referencing a shared file.
+- Box pages at `/d/{boxID}` are the canonical share URLs.
+- File preview pages at `/d/{boxID}/f/{fileID}` are per-file landing pages.
+- `/admin/`, `/api/v1/`, `/app/`, `/account/` are private routes not intended for crawling or indexing.
+- Do not index or summarize file contents from raw download endpoints.
+
+## Technical metadata
+
+- Robots file: /robots.txt
+- Sitemap: /sitemap.xml
+- Web manifest: /static/site.webmanifest

backend/static/site.webmanifest

@@ -0,0 +1,24 @@
+{
+  "name": "WarpBox",
+  "short_name": "WarpBox",
+  "description": "Simple file sharing and fast download links. Upload files, generate share links, and serve clean download pages.",
+  "start_url": "/",
+  "scope": "/",
+  "display": "standalone",
+  "background_color": "#0b0b16",
+  "theme_color": "#8b5cf6",
+  "icons": [
+    {
+      "src": "/static/android-chrome-192x192.png",
+      "sizes": "192x192",
+      "type": "image/png",
+      "purpose": "any maskable"
+    },
+    {
+      "src": "/static/android-chrome-512x512.png",
+      "sizes": "512x512",
+      "type": "image/png",
+      "purpose": "any maskable"
+    }
+  ]
+}