robots-txt

🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs

ScrapeGPT is a RAG-based Telegram bot designed to scrape and analyze websites, then answer questions based on the scraped content. The bot utilizes Retrieval Augmented Generation and webscraping to return natural language answers to the user's queries.