Best practices

  • Use explicit waits such as page.wait_for_selector(selector) to ensure elements are visible before interacting with them.

  • It's a good habit to specify a timeout in wait_for_selector to avoid indefinitely waiting for an element if it never becomes visible.

  • Utilize CSS selectors or XPath expressions in wait_for_selector to accurately target the specific elements you need to interact with.

  • Test visibility conditions on various screen sizes and resolutions to ensure your Playwright scripts are robust across different environments.

# pip install playwright
from playwright.sync_api import sync_playwright


with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto('https://sandbox.oxylabs.io/products')

    # Wait for an element to be visible using CSS selector
    page.wait_for_selector('.product-card h4', timeout=10)

    # Alternative: Wait for an element using XPath
    page.wait_for_selector('//*[contains(@class, "product-card")]//h4')

    # Perform actions after the element is visible
    print(page.query_selector('.product-card h4').text_content())
    browser.close()

Common issues

  • Ensure you're only processing content that's visually apparent to users and not obstructed by other elements or styles.

  • Verify that the page has fully loaded all dynamic content and scripts before attempting to wait for an element's visibility, as premature checks can lead to errors.

  • Consider using page.wait_for_function to check for more complex visibility conditions or states that are not directly supported by wait_for_selector.

  • Regularly update your selectors and visibility checks to adapt to changes in the web application's structure and design to maintain test reliability.

# Incorrect: Assuming element is visible without checking for obstructions
print(page.query_selector('.product-card h4').text_content())

# Correct: Ensure element is not only visible but also only visible to users
page.wait_for_selector('.product-card h4')
if page.evaluate(
    'element => window.getComputedStyle(element).opacity != "0"',
    page.query_selector('.product-card h4')
):
    print(page.query_selector('.product-card h4').text_content())


# Incorrect: Trying to interact with elements before dynamic content has loaded
print(page.query_selector('.product-card h4').text_content())

# Correct: Wait for all dynamic scripts and content to load before interaction
page.wait_for_load_state('networkidle')
print(page.query_selector('.product-card h4').text_content())


# Incorrect: Using outdated or incorrect selectors which may not reflect current page structure
page.query_selector('.old-product-class')

# Correct: Regularly update and verify selectors to match current web application structure
page.wait_for_selector('.product-card h4')
print(page.query_selector('.product-card h4').text_content())

Try Oyxlabs' Proxies & Scraper API

Residential Proxies

Self-Service

Human-like scraping without IP blocking

From

8

Datacenter Proxies

Self-Service

Fast and reliable proxies for cost-efficient scraping

From

1.2

Web scraper API

Self-Service

Public data delivery from a majority of websites

From

49

Useful resources

Playwright vs Puppeteer illustration
Playwright vs Puppeteer: The Differences
author avatar

Augustas Pelakauskas

2025-04-04

Advanced Web Scraping With Python Tactics in 2025
vytenis kaubre avatar

Vytenis Kaubrė

2025-01-01

How to Bypass CAPTCHA With Playwright
How to Bypass CAPTCHA With Playwright
author avatar

Yelyzaveta Hayrapetyan

2024-10-11

Get the latest news from data gathering world

I'm interested