Best practices

  • Use specific and unique CSS selectors to improve the accuracy and performance of element retrieval.

  • Opt for class selectors (`.class-name`) for elements with common styles, and ID selectors (`#id-name`) for unique elements, ensuring faster and more reliable element selection.

  • When selecting elements based on attributes, ensure to use the attribute value that uniquely identifies the element, like `[type='submit']` for buttons.

  • Combine CSS selectors to navigate DOM hierarchies, such as `.parent-class .child-class`, to target specific elements within a nested structure efficiently.

# pip install selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


# Initialize the WebDriver (using Chrome in this example)
driver = webdriver.Chrome()  # Make sure you have ChromeDriver installed


try:
    # Navigate to the webpage
    driver.get("https://sandbox.oxylabs.io/products")
    # Wait for page to load
    wait = WebDriverWait(driver, 10)
    
    # Select element by CSS class
    product_name = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".title")))
    print(product_name.text.strip())
    
    # Select element by element tag
    product_name = driver.find_element(By.TAG_NAME, "h4")
    print(product_name.text.strip())
    
    # Select elements by attribute (full class attribute)
    price = driver.find_element(By.CSS_SELECTOR, "[class='price-wrapper css-li4v8k eag3qlw4']")
    print(price.text.strip())
    
    # Combining CSS selectors
    info = driver.find_element(By.CSS_SELECTOR, ".category span")
    print(info.text.strip())
finally:
    # Close the browser
    driver.quit()

Common issues

  • Ensure your CSS selectors are updated to match any changes in the webpage's structure or style to avoid stale element reference errors.

  • Avoid using overly complex CSS selectors which can lead to slower execution times and harder maintenance; simplify where possible.

  • Use pseudo-classes like `:first-child`, `:last-child`, or `:nth-child(n)` to target specific elements in a list or among siblings for more precise control.

  • Regularly test and validate your CSS selectors in the browser's developer tools before implementing them in your Selenium scripts to catch any potential issues early.

# pip install selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait


driver = webdriver.Chrome()
driver.get("https://sandbox.oxylabs.io/products")
wait = WebDriverWait(driver, 10)


try:
    # Bad example: Using outdated CSS selector
    try:
        old_price = driver.find_element(By.CSS_SELECTOR, ".old-price-label")
        print(old_price.text.strip())
    except Exception as e:
        print(e)

    # Good example: Updated CSS selector after webpage changes
    current_price = driver.find_element(By.CSS_SELECTOR, ".price-wrapper")
    print(current_price.text.strip())


    # Bad example: Overly complex CSS selector
    complex_selector = driver.find_element(By.CSS_SELECTOR, "div.price-wrapper.css-li4v8k.eag3qlw4")
    print(complex_selector.text.strip())

    # Good example: Simplified CSS selector
    simple_selector = driver.find_element(By.CSS_SELECTOR, ".price-wrapper")
    print(simple_selector.text.strip())


    # Bad example: Selecting the first item without using pseudo-classes
    first_item = driver.find_elements(By.CSS_SELECTOR, ".category span")[0]
    print(first_item.text.strip())

    # Good example: Using :first-child pseudo-class
    first_item = driver.find_element(By.CSS_SELECTOR, ".category span:first-child")
    print(first_item.text.strip())


    # Bad example: Not testing CSS selector in browser before use
    try:
        untested_selector = driver.find_element(By.CSS_SELECTOR, ".unverified-selector")
        print(untested_selector.text.strip())
    except Exception as e:
        print(e)

    # Good example: Manually verify selector works in browser's developer tools first
    verified_selector = driver.find_element(By.CSS_SELECTOR, "h4")
    print(verified_selector.text.strip())


finally:
    driver.quit()

Try Oxylabs' Proxies & Scraper API

Residential Proxies

Self-Service

Human-like scraping without IP blocking

From

8

Datacenter Proxies

Self-Service

Fast and reliable proxies for cost-efficient scraping

From

1.2

Web scraper API

Self-Service

Public data delivery from a majority of websites

From

49

Useful resources

How to Find Elements With Selenium in Python
Enrika avatar

Enrika Pavlovskytė

2024-06-21

Web Scraping with Selenium and Python
Gabija Fatenaite avatar

Gabija Fatenaite

2024-06-13

XPath vs CSS Selectors
Monika Maslauskaite avatar

Monika Maslauskaite

2021-07-13

Get the latest news from data gathering world

I'm interested