BeautifulSoup

Note: old notes, from the AngularJS era of scraping. The recipes below still run (verified on 4.15.0) and BeautifulSoup itself is alive and well, but for today’s JavaScript-heavy sites an HTML parser alone rarely cuts it: you want a real browser (Playwright (Python)) or the site’s API.

References and links

• https://pypi.org/project/beautifulsoup4/
• https://www.crummy.com/software/BeautifulSoup/bs4/doc/

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')

Recipes, Tips and Tricks

How to remove comments from a bs4 element?

from bs4 import Comment

for comment in soup.find_all(string=lambda text: isinstance(text, Comment)):
    comment.extract()

The camelCase findAll and the text= argument still work but are deprecated (since 4.0); find_all and string= are the current spelling.

How to remove all the AngularJS bullshit attributes so you can actually read the HTML?

for tag in soup.descendants:
    if hasattr(tag, 'attrs'):
        tag.attrs = {
            key: value
            for key, value in tag.attrs.items()
            if not key.startswith('ng-')
        }

.descendants replaces the deprecated recursiveChildGenerator(). Note: this only removes the ng-* attributes, not the ng-* classes.

Verified on beautifulsoup4 4.15.0 with the lxml parser.