Register routes for robots.txt and sitemap.xml, and implement search engine indexing controls to protect user privacy.
Specifically:
- Set `X-Robots-Tag: noindex, nofollow, noarchive` headers on file downloads, thumbnails, and zip generation.
- Configure `Robots: web.RobotsNone` on download and preview pages to prevent indexing of temporary user uploads.
- Add canonical URLs, improved descriptions, and image alt tags to page metadata for better social sharing.
