How to Get a Web Element from HTML Source With Selenium

Web element

A web element is any HTML tag or part of a web page you can interact with – anything you find on a web page is a web element.

Some common web elements include:

• Clickable buttons

• Fillable Text fields

• Links that navigate to other pages

• Images that display visual content

• Dropdown menus

• Navigation bars

• Headers and footers

• Sliders and carousels

• Windows and popups

Web elements are defined using HTML code (for structure), styled with CSS (for appearance), and can be manipulated with JavaScript (for behavior). Identifying and working with these HTML elements is essential for creating interactive and functional websites in web development and testing.

Tutorial on how to retrieve a web element from an HTML source using Python

Prerequisites

• Python 3.6 or higher

• Selenium package

• Selenium Wire package for proxies

• Selenium WebDriver for your preferred browser

1. Installation

Install Selenium:

pip install selenium selenium-wire

Download the appropriate Selenium WebDriver for your browser:

• Chrome: ChromeDriver

• Firefox: GeckoDriver

• Edge: EdgeDriver

Let’s use ChromeDriver.

2. Proxy integration

Let's integrate proxies with Selenium to conceal your actual IP address and bypass CAPTCHA. Proxies enable you to spread HTTP requests across multiple proxy IPs, enhancing web scraping efficiency. They also allow you to access content that is restricted to specific regions.

As a rule, the best proxy providers are always paid services. Luckily, they tend to offer free proxies for testing purposes.

While Selenium enables proxies through the --proxy-server= parameter, it doesn't allow proxy authentication. For this reason, use Selenium Wire, an unofficial third-party package that extends Selenium's Python bindings and enables you to use authenticated proxies.

Let’s set up a Selenium WebDriver to use a proxy server and add Oxylabs Residential Proxies.

3. Web element retrieval

To find an element using Selenium and Python, use the get_attribute('outerHTML') or get_attribute('innerHTML') method.

The difference between outerHTML and innerHTML:

• outerHTML includes the element itself and all its content

• innerHTML includes only the content inside the element

Remember that the choice between outerHTML and innerHTML depends on your specific needs – whether you need the complete element with its tags or just the content within.

Similarly, various locators offer flexibility in targeting elements on a page:

• ID

• CSS selector

• XPath

• tag name

You can use different locators to find an element:

# Find by CSS selector
element = driver.find_element(By.CSS_SELECTOR, ".my-class")

# Find by XPath
element = driver.find_element(By.XPATH, "//div[@class='my-class']")

# Find by tag name
element = driver.find_element(By.TAG_NAME, "div")

Here's how to do it:

from seleniumwire import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service

# Proxy details
USER = "PROXY_USERNAME"
PASS = "PROXY_PASSWORD"
SERVER = "pr.oxylabs.io:7777"

# Create a driver with the proxy dict
driver = webdriver.Chrome(
    seleniumwire_options={
        "proxy": {
            "http": f"http://customer-{USER}:{PASS}@{SERVER}",
            "https": f"https://customer-{USER}:{PASS}@{SERVER}",
        }
    }
)

# Send a web request to a specific URL
driver.get("https://example.com")

# Find the element ID you want to get the HTML from
element = driver.find_element(By.ID, "my-element-id")  # or use other locators

# Get the outer HTML (including the element itself)
outer_html = element.get_attribute('outerHTML')
print("Outer HTML:", outer_html)

# Get the inner HTML (only the content inside the element)
inner_html = element.get_attribute('innerHTML')
print("Inner HTML:", inner_html)

# Close the driver
driver.quit()

NOTE: Using hardcoded credentials in the script is generally not recommended for security reasons.

Using other languages

Here’s the basic process of getting an HTML source using Java.

1. Initialize ChromeDriver

2. Navigate to the target page

3. Find an element with the ID my-element-id

4. Retrieve the outerHTML attribute

5. Retrieve the innerHTML attribute

6. Print both to the console

7. Close the WebDriver

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;

public class WebElementHtmlRetrieval {
    public static void main(String[] args) {
        // Initialize the WebDriver
        WebDriver driver = new ChromeDriver();
        driver.get("https://example.com");
        
        // Find the element ID
        WebElement element = driver.findElement(By.id("element-id"));
        
        // Get the outer HTML
        String outerHtml = element.getAttribute("outerHTML");
        System.out.println("Outer HTML: " + outerHtml);
        
        // Get the inner HTML
        String innerHtml = element.getAttribute("innerHTML");
        System.out.println("Inner HTML: " + innerHtml);
        
        // Close the driver
        driver.quit();
    }
}

Using different locators

// Find by CSS selector
WebElement elementByCSS = driver.findElement(By.cssSelector(".my-class"));
// Find by XPath
WebElement elementByXPath = driver.findElement(By.xpath("//div[@class='my-class']"));
// Find by tag name
WebElement elementByTag = driver.findElement(By.tagName("div"));

Wrap up

Whether you're building a web scraper, developing automated tests, or analyzing web content programmatically, understanding how to extract web elements from an HTML source is a fundamental skill.

Selenium provides methods like get_attribute('outerHTML') and get_attribute('innerHTML') that give you precise control over an HTML source you retrieve.

Alternatively, check competing web data extraction frameworks. See how they compare to Selenium:

• Playwright vs Selenium

• Scrapy vs Selenium

• Puppeteer vs Selenium