Selenium Proxy Integration
avatar

Iveta Vistorskyte

Oct 11, 2021 2 min read

Selenium is a tool that helps automate web browser interactions for website testing and more. It’s useful when you need to interact with a browser to perform a number of tasks, such as clicking on buttons, scrolling, etc. Even if primarily Selenium is used for website testing, it can also be used for web scraping because it helps locate the required public data on a website. 

This article will go through the Selenium integration process with Oxylabs’ Residential Proxies for a smooth web scraping process.

How to integrate Oxylabs’ proxies with Selenium?

This paragraph explains how to integrate Oxylabs’ Residential Proxies with Selenium in Python. Note that the required version of Python is Python 3.5 (or higher).

Setting up Selenium

Firstly, you’ll need to install Selenium Wire to extend Selenium’s Python bindings because using the default Selenium module for implementing proxies that require authentication makes the whole process complicated. You can do it using the pip command:

pip install selenium-wire

Another recommended package for this integration is webdriver-manager. It’s a package that simplifies the management of binary drivers for different browsers. In this case, there’s no need to manually download a new version of a web driver after each update.

You can install the webdriver-manager using the pip command as well:

pip install webdriver-manager

Proxy authentication

Once everything is set up, you can move on to the next part – proxy authentication. For proxies to work, you’ll be prompted to specify your account credentials.

USERNAME = "your_username"
PASSWORD = "your_password"
ENDPOINT = "pr.oxylabs.io:7777"

You’ll need to adjust your_username and your_password fields with the username and password of your proxy user.

Testing proxy server connection

If you need to check if the proxy is working, you should visit ip.oxylabs.io. If everything is working correctly, it will return an IP address of a proxy that you’re using.

try:
    driver.get("https://ip.oxylabs.io/")
    return f'\nYour IP is: {re.search(r"[0-9].{2,}", driver.page_source).group()}'
finally:
    driver.quit()

Full code for Oxylabs’ Residential Proxies integration with Selenium

import re
from typing import Optional

from seleniumwire import webdriver
# A package to have a chromedriver always up-to-date.
from webdriver_manager.chrome import ChromeDriverManager

USERNAME = "your_username"
PASSWORD = "your_password"
ENDPOINT = "pr.oxylabs.io:7777"


def chrome_proxy(user: str, password: str, endpoint: str) -> dict:
    wire_options = {
        "proxy": {
            "http": f"http://{user}:{password}@{endpoint}",
            "https": f"http://{user}:{password}@{endpoint}",
        }
    }

    return wire_options


def execute_driver():
    options = webdriver.ChromeOptions()
    options.headless = True
    proxies = chrome_proxy(USERNAME, PASSWORD, ENDPOINT)
    driver = webdriver.Chrome(
        ChromeDriverManager().install(), options=options, seleniumwire_options=proxies
    )
    try:
        driver.get("https://ip.oxylabs.io/")
        return f'\nYour IP is: {re.search(r"[0-9].{2,}", driver.page_source).group()}'
    finally:
        driver.quit()


if __name__ == "__main__":
    print(execute_driver())

Wrapping it up

Selenium is a great tool for public web scraping, especially when learning the basics. With the help of Oxylabs’ Residential Proxies, public web scraping becomes even more efficient. 

If you have any questions about integrating Oxylabs’ proxies, you can contact us at any time. You should also visit our GitHub profile for more integrations with Oxylabs’ Proxies.

avatar

About Iveta Vistorskyte

Iveta Vistorskyte is a Content Manager at Oxylabs. Growing up as a writer and a challenge seeker, she decided to welcome herself to the tech-side, and instantly became interested in this field. When she is not at work, you'll probably find her just chillin' while listening to her favorite music or playing board games with friends.

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.

Related articles

Oxylabs Proxy Integration With Ghost Browser

Oxylabs Proxy Integration With Ghost Browser

Oct 22, 2021

3 min read

Proxy Integration With WebHarvy

Proxy Integration With WebHarvy

Oct 22, 2021

3 min read

Playwright Proxy Integration With Oxylabs

Playwright Proxy Integration With Oxylabs

Oct 22, 2021

2 min read