meta: path: /examples/ai-integration name: AI/LLM Integration desc: "Make your documentation AI-friendly with automatic generation of llms.txt, TOON format, robots.txt, and SEO optimization" type: documentation format: toon/3.3 context: 7-page Dash docs site relatedPages[5]: Home: / Custom Directives: /examples/directives Data Visualization: /examples/visualization Getting Started: /getting-started Interactive .md: /examples/interactive sections[20]: 1. [3] Introduction This documentation boilerplate includes **AI/LLM integration** powered by [dash-improve-my-llms](https://pypi.org/project/dash-improve-my-llms/) v1.1.0. This feature automatically generates AI-friendly documentation, manages bot access, and optimizes your site for search engines. **New in v1.1. 2. [3] What Gets Generated The integration automatically creates several files that help AI assistants understand your application: 3. [4] 1. /llms.txt - AI-Friendly Documentation A markdown file optimized for Large Language Models that includes: - **Application overview** and purpose - **Interactive elements** (buttons, inputs, graphs) 4. [4] 2. /page.json - Technical Architecture A JSON file with technical details: ```json { 5. [4] 3. /architecture.txt - ASCII Overview A text-based visual representation of your application structure. Visit: [/architecture.txt](https://dash-documentation-boilerplate.onrender.com/architecture.txt) 6. [4] 4. /robots.txt - Bot Management Controls which bots can access your application: - ✅ **Allows**: AI search bots (ChatGPT-User, ClaudeBot, PerplexityBot) - ❌ **Blocks**: AI training bots (GPTBot, CCBot, anthropic-ai, Google-Extended) 7. [4] 5. /sitemap.xml - SEO Optimization An SEO-optimized sitemap with intelligent priority inference. Visit: [/sitemap.xml](https://dash-documentation-boilerplate.onrender.com/sitemap.xml) 8. [4] 6. /llms.toon - Token-Optimized Format (NEW in v1.0.0!) TOON (Token-Oriented Object Notation) format provides **50-60% fewer tokens** compared to markdown: ```toon meta: 9. [4] 7. /architecture.toon - Token-Optimized Architecture (NEW!) The architecture file in TOON format for reduced token usage: Visit: [/architecture.toon](https://dash-documentation-boilerplate.onrender.com/architecture.toon) --- 10. [3] TOON Format (Enhanced in v1.1.0) TOON (Token-Oriented Object Notation) is a token-optimized alternative to markdown that achieves **lossless semantic compression** - preserving all meaningful content while reducing tokens by **40-50%**. 11. [4] Design Principle > **TOON should be a LOSSLESS SEMANTIC COMPRESSION of llms.txt content** > > The goal is not maximum token reduction, but optimal information density. All meaningful content is preserved while removing only formatting overhead. 12. [4] Benefits | Format | Typical Size | Reduction | Best For | |--------|--------------|-----------|----------| | `llms.txt` | ~15,000 tokens | baseline | Human readability, full context | 13. [4] v1.1.0 TOON Enhancements The v1.1.0 release addresses 6 content gaps to provide complete information: **1. Application Context** - Related pages and multi-page awareness: ```toon 14. [4] Configure TOON Output ```python from dash_improve_my_llms import TOONConfig toon_config = TOONConfig( 15. [4] Generate TOON Programmatically ```python from dash_improve_my_llms import toon_encode, TOONConfig # Encode any Python data to TOON 16. [4] Test TOON Endpoints ```bash # Fetch TOON format (50-60% fewer tokens!) curl http://localhost:8553/llms.toon 17. [3] Configuration 18. [4] Base URL Configuration ```python # Set your production URL for proper sitemap generation app._base_url = "https://your-app-url.com" 19. [4] Bot Management ```python from dash_improve_my_llms import RobotsConfig app._robots_config = RobotsConfig( 20. [4] Page Metadata ```python from dash_improve_my_llms import register_page_metadata register_page_metadata( directives: llms_copy: [AI/LLM Integration] toc: [(none)] tables[2]: 1. [Format|Typical Size|Reduction|Best For] `llms.txt`|~15,000 tokens|baseline|Human readability, full context `llms.toon` v1.0.0|~200 tokens|98%|Too aggressive, lost content `llms.toon` v1.1.0|~6,000-8,000 tokens|40-50%|Lossless semantic compression `page.json`|Variable|-|Programmatic access, parsing 2. [Route|Purpose] `/llms.txt`|LLM-friendly documentation `/llms.toon`|Token-optimized docs (NEW!) `/page.json`|Technical architecture `/architecture.txt`|App overview `/architecture.toon`|Token-optimized architecture (NEW!) `/robots.txt`|Bot access control `/sitemap.xml`|SEO sitemap `//llms.txt`|Page-specific docs `//llms.toon`|Page-specific TOON (NEW!) keyLists[5]: - unordered: **Application overview** and purpose, **Interactive elements** (buttons, inputs, graphs), **Data flow** and callback descriptions, **Component hierarchy** and relationships, **Key features** and capabilities - unordered: ✅ **Allows**: AI search bots (ChatGPT-User, ClaudeBot, Perpl..., ❌ **Blocks**: AI training bots (GPTBot, CCBot, anthropic-ai,..., ✅ **Allows**: Traditional search engines (Googlebot, Bingbot... - unordered: LLMs recognize these as key interactive elements, Appears prominently in llms.txt, Helps AI assistants guide users more effectively - unordered: Your app's purpose and capabilities, Interactive components and their IDs, Data flow and callback logic, How to use different features, Component relationships - unordered: **GPTBot** (OpenAI), **CCBot** (Common Crawl), **anthropic-ai** (Anthropic), **Google-Extended** (Google AI training), **FacebookBot** (Meta AI) tips[17]: /page.json - Technical Architecture: [json] {... /llms.toon - Token-Optimized Format (NEW in v1.0.0!): [toon] meta:... v1.1.0 TOON Enhancements: [toon] context: Part of multi-page Dash app with 3 total pages... v1.1.0 TOON Enhancements: [toon] purpose:... Test TOON Endpoints: [bash] # Fetch TOON format (50-60% fewer tokens!)... Base URL Configuration: [python] # Set your production URL for proper sitemap generation... Bot Management: [python] from dash_improve_my_llms import RobotsConfig... Page Metadata: [python] from dash_improve_my_llms import register_page_metadata... Hiding Sensitive Pages: [python] from dash_improve_my_llms import mark_hidden... Hiding Components: [python] from dash_improve_my_llms import mark_component_hidden... Structured Data: [json] {... Meta Tags: [html] ... Noscript Fallback: [html]