Building an AI Web Crawler That Outputs LLM-Ready Content Chunks
Building an AI Web Crawler That Outputs LLM-Ready Content Chunks The biggest challenge in building RAG (Retrieval-Augmented Generation) pipelines isn't the vector database or the embeddings — it's getting clean, structured content from web pages in the first place. Raw HTML is noisy, full of navigation, ads, and boilerplate that pollutes your embeddings. I built an AI Web Crawler that extracts clean, chunked content from any website — ready to feed directly into LLMs and RAG systems. When you fe
Building an AI Web Crawler That Outputs LLM-Ready Content Chunks
The biggest challenge in building RAG (Retrieval-Augmented Generation) pipelines isn't the vector database or the embeddings — it's getting clean, structured content from web pages in the first place. Raw HTML is noisy, full of navigation, ads, and boilerplate that pollutes your embeddings.
I built an AI Web Crawler that extracts clean, chunked content from any website — ready to feed directly into LLMs and RAG systems.
The Problem with Web Content for AI
When you feed raw HTML to an LLM, you waste tokens on navigation menus, cookie banners, and footer links. When you feed it poorly cleaned text, retrieval quality drops because the chunks don't align with semantic boundaries.
The solution: a crawler that understands document structure and outputs content chunks designed for AI consumption.
Key Features
Smart Content Extraction
The crawler strips boilerplate (nav, footer, scripts, styles) and extracts only the main content — the actual article, documentation, or page text that matters.
Semantic Chunking
Content is split into chunks that respect heading boundaries:
def chunk_content(text, headings, max_chunk=500):
chunks = []
current = []
for heading in headings:
section = extract_section(text, heading)
if len(section) > max_chunk:
chunks.extend(split_long(section, max_chunk))
else:
chunks.append(section)
return chunks
Enter fullscreen mode Exit fullscreen mode
Token Estimation
Every chunk includes an estimated token count, so you know exactly how much context you're working with before sending to your LLM:
tokens = len(chunk_text) // 4 # rough estimate
Enter fullscreen mode Exit fullscreen mode
Structured Data Extraction
Beyond plain text, the crawler captures:
- Headings — document structure/hierarchy
- Tables — structured data preserved as markdown
- Internal/External Links — relationship mapping
- JSON-LD — structured data from schema.org
- OpenGraph — social metadata
Use Cases
- RAG Pipelines: Feed chunks directly to your vector database
- Knowledge Base Building: Crawl documentation sites systematically
- Content Monitoring: Track changes on competitor pages
- Dataset Creation: Build training data from web sources
- AI Agent Tools: Give your autonomous agents a web reader
Try It
- Apify Store: AI Web Crawler
- RapidAPI: Multi-Tool Content API
- GitHub: multi-tool-content-api
The Apify actor uses pay-per-event pricing — you only pay for pages successfully crawled.
Conclusion
The future of web scraping is AI-ready output. By chunking content semantically, estimating tokens, and preserving structure, we bridge the gap between raw web pages and LLM-ready knowledge.
Built with Python, BeautifulSoup4, httpx, and the Apify platform.
