Selenium (Python)

Note: older content, kept for the archive. For new browser automation work I reach for Playwright (Python) instead: auto-waiting removes most of the retry code you write by hand here. The recipes below still work, and were rerun on Selenium 4.51.0 driving Chrome 155.

The snippets used to be written in the Selenium 3 spelling, browser.find_element_by_id("x"). Those methods were removed in Selenium 4.3.0 (June 2022) and raise AttributeError on any current version, so they have been rewritten with By:

from selenium.webdriver.common.by import By

el = driver.find_element(By.ID, "username")

One other thing has changed for the better: since 4.6 there is nothing to install by hand. Selenium Manager resolves and downloads a matching driver on its own, so this is a complete working start:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

References and links

• selenium-python.readthedocs.io
• Selenium Manager for driver resolution

Recipes, tips and tricks

How to get the HTML of one element?

element.get_attribute("innerHTML")

This reads the DOM property, so it returns the markup even for an element that is hidden with display: none.

How to get the next sibling of an element?

el = driver.find_element(By.XPATH, "//table[@id='t']/tbody/tr[1]")
next_el = el.find_element(By.XPATH, "following-sibling::*[1]")

How to fill in a login / password form (or any other form)?

driver.find_element(By.ID, "username").send_keys("thatsme")
driver.find_element(By.ID, "password").send_keys("s3cr3t")
driver.find_element(By.ID, "signin-form").submit()

submit() on any element inside the form submits that form, which saves you from hunting for the right button. It is still there in Selenium 4.

How to wait for content that JavaScript renders?

There is no implicit wait by default, so reading an element straight after get() gives you whatever was in the initial HTML. On a page whose text is filled by a timer, .text returned the placeholder; the same read behind an explicit wait returned the real value:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 5).until(
    EC.text_to_be_present_in_element((By.ID, "status"), "ready")
)
el = driver.find_element(By.ID, "status")

Mind what until() hands back: it returns whatever the condition returned, and text_to_be_present_in_element returns True, not the element. Conditions like presence_of_element_located do return the element.

Prefer WebDriverWait over driver.implicitly_wait(...): an implicit wait applies to every lookup in the session, including the ones you expect to fail, and mixing the two gives unpredictable timeouts.

How to make sure a given element is visible?

Some frontend frameworks refuse clicks on an element that is not in view. Scrolling it into view and then back up a little keeps it clear of a sticky header:

el = driver.find_element(By.ID, "signup")
driver.execute_script("arguments[0].scrollIntoView(true); window.scrollBy(0, -120);", el)

el.is_displayed() tells you whether the element is rendered at all, which is a different question from whether it is in the viewport.

Does the site know it is a robot?

Yes, if it looks. In headless Chrome, navigator.webdriver is true and the user agent contains HeadlessChrome. Both are visible to any script on the page.