Best practices

  • Ensure that the web page is fully loaded before attempting to retrieve text to avoid `NoSuchElementException`.

  • Use explicit waits to handle elements that might take longer to appear due to dynamic content loading.

  • Always validate the correctness of the locator (ID, class name, XPath) to ensure it uniquely identifies the target element.

  • Keep your Selenium drivers updated to ensure compatibility with the latest versions of web browsers.

# Importing necessary libraries
from selenium import webdriver
from selenium.webdriver.common.by import By

# Setting up the WebDriver
driver = webdriver.Chrome()

# Opening the target webpage
driver.get("https://sandbox.oxylabs.io/products")

# Example 1: Get text using element ID
element_text_by_id = driver.find_element(By.ID, "element-id").text
print(element_text_by_id)

# Example 2: Get text using element class name
element_text_by_class = driver.find_element(By.CLASS_NAME, "element-class").text
print(element_text_by_class)

# Example 3: Get text using XPath
element_text_by_xpath = driver.find_element(By.XPATH, "//tagname[@attribute='value']").text
print(element_text_by_xpath)

# Closing the driver
driver.quit()

Common issues

  • Ensure that the element is visible on the page before attempting to retrieve its text, as hidden elements might return an empty string.

  • Check for iframes and switch to the appropriate iframe context if the element is within one, as Selenium does not automatically manage iframe switches.

  • Handle potential `StaleElementReferenceException` by re-locating the element or using a retry mechanism in your script.

  • Consider using `get_attribute('textContent')` instead of `.text` if you need to retrieve text that includes child element texts.

# Incorrect: Trying to get text from a hidden element
hidden_text = driver.find_element(By.ID, "hidden-element").text

# Correct: Check if element is visible before getting text
element = driver.find_element(By.ID, "visible-element")
if element.is_displayed():
visible_text = element.text

# Incorrect: Ignoring iframe and trying to access element directly
text_inside_iframe = driver.find_element(By.ID, "iframe-element").text

# Correct: Switch to the iframe first then access the element
driver.switch_to.frame("iframe-name")
text_inside_iframe = driver.find_element(By.ID, "iframe-element").text
driver.switch_to.default_content()

# Incorrect: Directly accessing an element that might be stale
stale_text = driver.find_element(By.ID, "dynamic-element").text

# Correct: Use a try-except block to handle StaleElementReferenceException
try:
dynamic_text = driver.find_element(By.ID, "dynamic-element").text
except StaleElementReferenceException:
dynamic_text = driver.find_element(By.ID, "dynamic-element").text

# Incorrect: Using .text on an element with nested child texts
parent_text = driver.find_element(By.ID, "parent-element").text

# Correct: Use get_attribute('textContent') to include child element texts
complete_text = driver.find_element(By.ID, "parent-element").get_attribute('textContent')

Try Oxylabs' Proxies & Scraper API

Residential Proxies

Self-Service

Human-like scraping without IP blocking

From

8

Datacenter Proxies

Self-Service

Fast and reliable proxies for cost-efficient scraping

From

1.2

Web scraper API

Self-Service

Public data delivery from a majority of websites

From

49

Useful resources

Get the latest news from data gathering world

I'm interested