LLMs.txt Generator (Alpha) - A tool for converting website content into structured text files for AI model training.

## Primary Function of LLMs.txt Generator The primary function of the LLMs.txt Generator (Alpha) is to convert website content into structured text files (llms.txt for concise summaries and llms-full.txt for complete text) that are directly usable by AI models, particularly for training or analyzing large language models (LLMs). ## Output Files of LLMs.txt Generator The LLMs.txt Generator produces two types of files: 1. **llms.txt**: A concise summary of the website content. 2. **llms-full.txt**: The complete text extracted from the website. ## Technologies in LLMs.txt Generator v2 The LLMs.txt Generator v2 reportedly achieves a 10x speed improvement through optimized algorithms and asynchronous processing techniques. However, this claim is not explicitly documented in official sources and may require further verification. ## Configurable Parameters in LLMs.txt Generator The tool allows configuration via the following parameters: - **maxUrls**: Controls the number of URLs to crawl (range: 1–100; default: 10). - **showFullText**: Determines whether to generate the full text output (default: false). ## Billing and Usage Limits - **Billing**: Each URL processed costs 1 credit point. - **Limits**: - Only public pages are supported. - The alpha version allows processing up to 5,000 URLs. ## Asynchronous Processing in LLMs.txt Generator The tool supports asynchronous processing through the following functions: 1. **async_generate_llms_text**: Creates an asynchronous task for crawling and generating files. 2. **check_generate_llms_text_status**: Monitors the task status and returns generated files along with the count of processed URLs. ## Official Documentation The official documentation for the LLMs.txt Generator is available at [Firecrawl's documentation page](https://docs.firecrawl.dev/features/alpha/llmstxt). Additional resources include: - [Firecrawl Blog](https://www.firecrawl.dev/blog/How-to-Create-an-llms-txt-File-for-Any-Website) - [GitHub Repository](https://github.com/mendableai/llmstxt-generator) ## Initialization Requirements To use the LLMs.txt Generator, users must: 1. Provide an API key (e.g., `firecrawl = FirecrawlApp(api_key="your_api_key")`). 2. Configure parameters like `maxUrls` and `showFullText` as needed. ## Feedback and Support Users can contact the support team via email at [[email protected]](mailto:[email protected]) for feedback or assistance. ### Citation sources: - [LLMs.txt Generator (Alpha)](https://docs.firecrawl.dev/features/alpha/llmstxt) - Official URL Updated: 2025-04-01