What problem does it solve?
Analyzing a large folder of downloaded web pages manually is slow and unstructured. WebMiner groups similar HTML pages into clusters and produces an interactive HTML report plus Excel spreadsheets, all locally with no LLM tokens consumed.
Core Features & Use Cases
- Three-Stage Local Pipeline: Encodes each HTML page into a 69-dimension feature vector, clusters pages with SMILE KMeans (k auto-detected), and renders interactive views.
- Interactive Reports and Spreadsheets: Generates an index.html report for exploring clusters plus .xlsx files for sorting, filtering, and further analysis in Excel.
- Resumable Runs: Supports --resume to continue interrupted pipelines and --max-files to limit corpus size.
- Use Case: You have crawled 500 product detail pages from an e-commerce site. Run the full pipeline to cluster them by page structure, then open the generated report to inspect each cluster and export results to Excel.
Quick Start
Run browser4-cli webminer install once, then ask the assistant to run browser4-cli webminer all on your folder of downloaded HTML files.
