Links tagged “crawler”

vincentschmalbach.com

same URL, same second: 303 KB for Chrome, 13 KB of markdown for ClaudeBot, a 406 for GPTBot.

github.com

A catalogue of technical and non-technical measures that make automated scraping harder — rate limiting, honeypots, shifting markup, JavaScript-loaded content — and what each costs real users.

github.com

A Python rewrite of the Goose article extractor, pulling the main text, metadata, images and embedded video out of news pages.

github.com

A Python module for web mining, bundling crawling and scraping, natural language processing, machine learning, network analysis and visualisation.

github.com

Book chapter by A. Jesse Jiryu Davis and Guido van Rossum building a concurrent web crawler three ways — threads, then callbacks, then generator-based coroutines.

docs.python.org

Python 2's robotparser module and its RobotFileParser class, which reads robots.txt and answers whether a user agent may fetch a URL.

outwit.com

OutWit Hub, a desktop application for Windows, macOS and Linux that finds, extracts and organises data and media from web pages.

arthurdejong.org

Download page for webcheck, an extensible site-checking tool for webmasters, listing release tarballs with signatures plus Git repository instructions.