Add view as markdown button

[Marketen]

This pull request introduces a new system to generate and expose plain markdown versions of all documentation pages, optimized for LLM consumption. It adds a custom Docusaurus plugin to automate markdown extraction, creates an index file of all markdown URLs, and enhances the documentation UI with a "View as Markdown" button for direct access. Additional improvements include dependency updates and SEO enhancements.

An example of LLM markdown pages in beaconcha.in:

• normal page (for humans): https://docs.beaconcha.in/api/overview
• .md page (for llms): https://docs.beaconcha.in/api/overview.md

---

PR implementation:

In each markdown file, there is now some metadata (title and a little description about the file). This is a docusaurus native functionality that we can use when generating the llms.txt index
https://docusaurus.io/docs/markdown-features/head-metadata

SEO/Robots

• Added a robots.txt file allowing all crawlers and referencing the sitemap, improving site SEO and discoverability. It makes crawlers ignore the .md files taht are for LLMs only

[netlify]

✅ Deploy Preview for steady-daffodil-d78dee ready!

Name	Link
🔨 Latest commit	502c4f3
🔍 Latest deploy log	https://app.netlify.com/projects/steady-daffodil-d78dee/deploys/6978b9862059f500088f86c3
😎 Deploy Preview	https://deploy-preview-505--steady-daffodil-d78dee.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[github-actions]

🚀 Deployed on https://6978b9dbc005c698b1960001--steady-daffodil-d78dee.netlify.app