crawler

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

A community-driven way to read and chat with AI bots - powered by chatGPT.

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Web Application Security Scanner Framework

Headless Chrome .NET API

All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers

https://spatie.be/docs/crawler

:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Get web data for AI agents and LLMs

Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

Google, Naver multiprocess image web crawler (Selenium)

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.