Vibe Coding Discoverdoc-scraper
View on GitHubGo web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data).
A configurable Go crawler that converts documentation sites to structured Markdown and JSONL for LLM ingestion. It also offers offline BM25 search and an MCP server interface for clients such as Claude Code and Cursor.
Use Cases
Crawl technical documentation into clean Markdown for RAGExport crawled pages as JSONL for ingestion pipelinesSearch downloaded documentation offline with BM25Expose documentation crawling and search to MCP clients
Built With
- Language
- Go
- Frameworks
- Model Context Protocol · SQLite FTS5 · BadgerDB
Tags
documentation crawler · web scraping · RAG ingestion · Markdown conversion · MCP server · offline search · BM25 · JSONL · incremental crawling · Go CLI