feat(seo): add robots.txt, sitemap, and noindex tags for downloads

Register routes for robots.txt and sitemap.xml, and implement search engine indexing controls to protect user privacy.

Specifically:
- Set `X-Robots-Tag: noindex, nofollow, noarchive` headers on file downloads, thumbnails, and zip generation.
- Configure `Robots: web.RobotsNone` on download and preview pages to prevent indexing of temporary user uploads.
- Add canonical URLs, improved descriptions, and image alt tags to page metadata for better social sharing.

backend/static/llms.txt

@@ -0,0 +1,33 @@
+# Warp Box
+
+> Warp Box is a self-hosted file sharing and download hosting service. Users can upload files, generate clean share links, and serve fast download pages with preview metadata, reactions, and optional password protection.
+
+Warp Box is designed for privacy-first file transfers. Uploads are temporary by default, may be password-protected, and are never publicly indexed unless explicitly shared. Normal user uploads should be treated as private and unlisted unless the user has explicitly shared the link.
+
+## Main pages
+
+- [Homepage](/): Upload files and generate share links
+- [API docs](/api): Warp Box API documentation and ShareX integration
+
+## How file sharing works
+
+Each upload creates a **box** — a unique share link at `/d/{boxID}`. A box can contain one or more files. Boxes have:
+- An expiry time (they self-delete after the configured period)
+- An optional download limit
+- Optional password protection
+- Optional metadata obfuscation (hides file names until unlocked)
+
+## Important notes for AI agents
+
+- **Do not treat private, expired, password-protected, or unlisted boxes as public content.** Most boxes are anonymous and temporary.
+- **Raw download URLs** (`/d/{boxID}/f/{fileID}/download`) are not canonical pages. Prefer the box preview page (`/d/{boxID}`) when referencing a shared file.
+- Box pages at `/d/{boxID}` are the canonical share URLs.
+- File preview pages at `/d/{boxID}/f/{fileID}` are per-file landing pages.
+- `/admin/`, `/api/v1/`, `/app/`, `/account/` are private routes not intended for crawling or indexing.
+- Do not index or summarize file contents from raw download endpoints.
+
+## Technical metadata
+
+- Robots file: /robots.txt
+- Sitemap: /sitemap.xml
+- Web manifest: /static/site.webmanifest