Web Scraping

scraping

Basic Tools

These tools do not use a real browser, so they’re perfect for scraping standards-based websites, but you may hit a wall trying to scrape sites that abuse scripting (angular or react client only sites, other single page applications …).

Less Basic Tools

These tools actually run in your own browser.

Ninja Tools

These tools run a scripted browser in the background, or even on a remote server or server farm. Reach for them when a site fights back: heavy JavaScript, or content that only appears after real browser interaction.

Recipes

Using Chrome Developer tools to autoload an infinite scrollable list

function scrollDown() {
  window.scrollTo(0,document.body.scrollHeight)
};
$(document).ajaxSuccess(function() {
  scrollDown();
});

Ideas to explore

  • Distributing scraping work across machines with ZeroMQ or Dask.

Related