Best practices

  • Use explicit waits (e.g., WebDriverWait) to handle dynamic content that might not be immediately available when the page loads.

  • Always validate your XPath expressions using browser developer tools before integrating them into your scripts.

  • Be cautious with contains() in XPath – it can unintentionally match multiple elements. Refine your expressions to be as specific as possible.

  • Keep your Selenium WebDriver and browser updated to maintain compatibility and leverage new features and bug fixes.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Initialize WebDriver
driver = webdriver.Chrome()

# Navigate to the page
driver.get("https://sandbox.oxylabs.io/products")

# Method 1: Using XPath to find element by exact text
element = driver.find_element(By.XPATH, "//tagname[text()='Exact Text']")

# Method 2: Using XPath with contains() for partial text match
partial_element = driver.find_element(By.XPATH, "//tagname[contains(text(), 'Part of Text')]")

# Method 3: Using WebDriverWait to find element by text with timeout
wait_element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, "//tagname[contains(text(), 'Timely Text')]"))
)

# Close the browser
driver.quit()

Common issues

  • Ensure that the text used in XPath expressions is not dynamically generated, as this can lead to element not found errors.

  • Avoid using absolute XPaths; instead, opt for relative paths to increase the robustness and maintainability of your tests.

  • Overusing text() can reduce flexibility – attributes (like id or class) are often more stable.

  • Text matches are case-sensitive in XPath. If case may vary, use translate() for a case-insensitive match.


# Incorrect: Using dynamically generated text in XPath
element = driver.find_element(By.XPATH, "//div[text()='Session ID: 12345']")

# Correct: Use static parts of the text or attributes
element = driver.find_element(By.XPATH, "//div[contains(text(), 'Session ID:')]")

# Incorrect: Using absolute XPath
element = driver.find_element(By.XPATH, "/html/body/div[1]/div[2]/div[3]/div")

# Correct: Using relative XPath
element = driver.find_element(By.XPATH, "//div[@class='specific-class']")

# Incorrect: Overusing text() where attributes could be used
element = driver.find_element(By.XPATH, "//button[text()='Click me']")

# Correct: Targeting attributes
element = driver.find_element(By.XPATH, "//button[@id='submit-button']")

# Incorrect: Ignoring case sensitivity in text searches
element = driver.find_element(By.XPATH, "//div[text()='welcome']")

# Correct: Using translate() to handle case variations
element = driver.find_element(By.XPATH, "//*[translate(text(), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz')='welcome']")

Try Oxylabs' Proxies & Scraper API

Residential Proxies

Self-Service

Human-like scraping without IP blocking

From

8

Datacenter Proxies

Self-Service

Fast and reliable proxies for cost-efficient scraping

From

1.2

Web scraper API

Self-Service

Public data delivery from a majority of websites

From

49

Useful resources

Puppeteer vs Selenium: Which to Choose
author avatar

Yelyzaveta Hayrapetyan

2025-05-27

How to Find Elements With Selenium in Python
Enrika avatar

Enrika Pavlovskytė

2024-06-21

Web Scraping with Selenium and Python
Gabija Fatenaite avatar

Gabija Fatenaite

2024-06-13

Get the latest news from data gathering world

I'm interested