Batch-scrape Hundreds Of Urls Concurrently

Rust web-crawling engine that turns sites into clean Markdown and structured metadata, with 14 language bindings plus an MCP server and agent skills for feeding AI agents and RAG pipelines. SSRF-safe, robots-aware, optional headless-browser rendering.

AI Tools | Rust · web-crawling · web-scraping