scraping

The context API to search, scrape, and interact with the web at scale. 🔥

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

Scrapy, a fast high-level web crawling & scraping framework for Python.

🕵️‍♂️ Collect a dossier on a person by username from 3000+ sites

Open source AI job application bot in Python: browser automation and web scraping to read job postings, then auto-apply with a tailored resume and cover letter for each posting.

Python scraper based on AI

Elegant Scraper and Crawler Framework for Golang

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Pythonic HTML Parsing for Humans™

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

A scalable web crawler framework for Java.

Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Mechanize is a ruby library that makes automated web interaction easy.

Collection of useful data science topics along with articles, videos, and code

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

A browser testing and web crawling library for PHP and Symfony

Get web data for AI agents and LLMs

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.