How to wait for page load in Selenium?

Learn how to effectively wait for a page to fully load using Selenium in this concise tutorial. Master synchronization techniques to ensure accurate data extraction and enhance your scraping efficiency.

Best practices

Use implicit waits to handle scenarios where elements take time to load, but avoid using them as a default method due to potential increased test execution time.
Employ explicit waits to target specific elements or conditions, enhancing test reliability by ensuring elements are interactable before proceeding.
Utilize JavaScript execution within explicit waits to check for the 'complete' readyState of the document, ensuring the page has fully loaded before actions are performed.
Always define a reasonable timeout for explicit waits to prevent indefinitely hanging tests if the expected condition is never met.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

# Initialize the WebDriver
driver = webdriver.Chrome()

# Navigate to the target URL
driver.get("https://sandbox.oxylabs.io/products")

# Method 1: Implicit wait
driver.implicitly_wait(10) # Waits up to 10 seconds for elements to become available

# Method 2: Explicit wait for a specific element
WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.ID, "specificElementId")))

# Method 3: Wait until a page is completely loaded
WebDriverWait(driver, 30).until(lambda driver: driver.execute_script('return document.readyState') == 'complete')

# Close the browser
driver.quit()

Common issues

Ensure that the element locators used in explicit waits are updated and accurate to avoid NoSuchElementException.
Adjust the timeout settings based on network speed and page complexity to optimize waiting times without causing unnecessary delays.
Consider using WebDriverWait in combination with expected_conditions like visibility_of_element_located to handle AJAX-loaded elements more effectively.
Regularly review and update your waiting strategies to adapt to changes in web application behavior and structure, ensuring robustness in automated tests.

# Incorrect: Using a fixed ID that may change or be dynamic
WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.ID, "oldElementId")))

# Correct: Ensure the locator is current and reflects the element's latest attributes
WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.ID, "updatedElementId")))

# Incorrect: Setting a very short timeout for a complex page, leading to TimeoutException
driver.implicitly_wait(2)

# Correct: Adjusting the timeout to a reasonable duration considering page complexity
driver.implicitly_wait(15)

# Incorrect: Using presence_of_element_located for elements that are there but not visible
WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".hidden-element")))

# Correct: Using visibility_of_element_located to wait for elements that need to be interacted with
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".visible-element")))

# Incorrect: Sticking to an old waiting strategy even after the application's loading behavior has changed
driver.implicitly_wait(10)

# Correct: Regularly updating waiting strategies to align with current web application behavior
WebDriverWait(driver, 30).until(lambda driver: driver.execute_script('return document.readyState') == 'complete')