How to Set Up Proxies With AIOHTTP: HTTP, SOCKS5 & Rotating Proxies

Prerequisites

Beforehand, please ensure that you have the following prerequisites in place: 

• Python installed (3.6 or above)

• Oxylabs sub-user’s credentials

Installing AIOHTTP

Run the following pip command in the terminal to install the aiohttp library:

pip install aiohttp

Use the following command in a Windows command prompt:

python -m pip install aiohttp

Additionally, you must install the asyncio package as it provides many tools for handling multiple tasks efficiently. For installation, use the following command:

pip install asyncio

Now, the aiohttp library is all set to send web requests and receive web responses. The next step is to import the required packages in the Python file:

import aiohttp
import asyncio

Handling HTTP requests using AIOHTTP

The following example shows how to use aiohttp to handle HTTP protocol-based requests. We’ll send an HTTP GET request to the https://ip.oxylabs.io/ web page. After receiving the request, the web page returns the IP address of the requester. 

Let’s start by creating the get_response() async function:

async def get_response():
    async with aiohttp.ClientSession() as session:
        async with session.get(
                'https://ip.oxylabs.io/'
        ) as response:
            print('Status Code: ', response.status)
            print('Body: ', await response.text())

When executed, the get_response() function creates a session by using the ClientSession() method and then sends an HTTP GET request to the target URL. Once the response to this GET request arrives, the function prints the status code value and the response text. 

Note: You can also configure request timeouts in aiohttp to prevent requests from waiting indefinitely for a response.

Let’s call this get_response() to see the HTTP request in action:

asyncio.run(get_response())

Proxy integration using basic authentication 

In the previous example, you can see that we’ve used the username, password, and proxy address in a single to perform proxy authentication with the GET request. Aiohttp also provides us with another way to perform user authentication – the BasicAuth method. The following example uses the BasicAuth method to pass the username and password with the request.

import asyncio
import aiohttp

PROXY_ADDRESS = 'proxy_address'
USERNAME = 'username'
PASSWORD = 'password'

async def get_response_using_proxy():
    async with aiohttp.ClientSession() as session:
        async with session.get(
                'https://ip.oxylabs.io/',
                proxy=f'http://{PROXY_ADDRESS}',
                proxy_auth=aiohttp.BasicAuth(USERNAME, PASSWORD)
        ) as response:
            print('Status Code: ', response.status)
            print('Body: ', await response.text())

asyncio.run(get_response_using_proxy())

How to rotate proxies with Python and AIOHTTP

Some websites restrict web data access if certain IP addresses raise suspicion, which can lead to failed requests. There’s a possibility your access will be restricted if you use a single IP for repetitive scraping requests.

Oxylabs’ Residential Proxies can randomly change the proxy server address with each request or maintain the same IP through a sticky session for up to 24 hours. These options are also available with Shared Datacenter Proxies, although an IP can be retained indefinitely. As a result, proxy rotation is performed internally by Oxylabs’ Residential and Shared Datacenter Proxies, so you don't need to take care or rotating proxies externally. For additional information, see our documentation for Residential and Shared Datacenter Proxies.

On the other hand, Dedicated Datacenter Proxies don’t offer an inherent rotation functionality, but you can use our Proxy Rotator to quickly construct that.

As an alternative, Python aiohttp can be used for proxy rotation. Although the library has no built-in rotation feature, you may rotate proxy servers using the following two techniques.

Selecting a random proxy from the proxy list

The most straightforward way to rotate proxies is to have a proxy list and randomly select any of the proxies from it.

If you have a list of proxies, the following code will allow you to rotate them for every web request:

import asyncio
import random
import aiohttp

proxy_list = [
    'http://username:passwordW@PROXY_ADDRESS_1:60000',
    'http://username:password@PROXY_ADDRESS_2:60000',
                    .
                    .
                    .
    'http://username:password@PROXY_ADDRESS_N:60000'
]

proxy = random.choice(proxy_list)

async def get_response_using_proxy():
    async with aiohttp.ClientSession() as session:
        async with session.get(
                'https://ip.oxylabs.io/',
                proxy=proxy
        ) as response:
            print('Status Code: ', response.status)
            print('Body: ', await response.text())

asyncio.run(get_response_using_proxy())

The above code creates a list of different IP addresses with usernames and passwords. Then it uses the random.choice() method to randomly select one proxy endpoint from the proxy_list. Once the proxy is fetched from the list of proxies randomly, the request is sent using this selected proxy server endpoint.

The random.choice() method can select the same proxy endpoints multiple times, so there’s a chance to use a particular proxy address for multiple requests.

Iterating over a proxy list

The previous proxy rotation approach is non-deterministic due to its unpredictability. Let’s use a more predictable rotation approach instead by employing a round-robin-style strategy.

With this method, you’ll build a list of proxy endpoints and cycle over the list indices until the list is finished. The subsequent value of i is mapped to the 0th index using modular arithmetic. It continues until all iterations of the for loop have been completed.

The following code demonstrates this concept:

import asyncio
import aiohttp

proxy_list = [
   'http://username:password@PROXY_ADDRESS_1:60000',
   'http://username:password@PROXY_ADDRESS_2:60000',
                .
                .
                .
   'http://username:password@PROXY_ADDRESS_N:60000'
]

async def get_response_using_proxy(target_url, proxy):
    async with aiohttp.ClientSession() as session:
        async with session.get(
                target_url,
                proxy=proxy,
        ) as response:
            print('Status Code: ', response.status)
            print('Body: ', await response.text())

number_of_requests = 10
length = len(proxy_list)
for i in range(number_of_requests):
    index = i % length
    asyncio.run(get_response_using_proxy('https://ip.oxylabs.io/', proxy_list[index]))

The for loop sends HTTP requests equal to the number_of_requests. Note how the index is determined throughout each cycle. The index value is kept within the proxy's boundaries using the expression i%length. It maps all i values in the range of 0 and length-1, resulting in a smooth proxy rotation.

How to reuse proxies

With proxy rotation, we select a different proxy on each request sent by aiohttp. We can reuse the same proxy until the access gets interrupted and then use another one. 

Let’s see an example:

import asyncio
import aiohttp

proxy_list = [
    'http://username:password@PROXY_ADDRESS_1:60000',
    'http://username:password@PROXY_ADDRESS_2:60000',
                    .
                    .
                    .
    'http://username:password@PROXY_ADDRESS_N:60000'
]

async def get_response_using_proxy(target_url, proxy):
    async with aiohttp.ClientSession() as session:
        async with session.get(
                target_url,
                proxy=proxy
        ) as response:
            print('Status Code: ', response.status)
            print('Body: ', await response.text())
            return response.status

index = 0
number_of_requests = 10
length = len(proxy_list)
for _ in range(number_of_requests):
    status_code = asyncio.run(get_response_using_proxy('https://ip.oxylabs.io/location', proxy_list[index]))
    if status_code != 200:
        index = index + 1  # selecting new proxy index
        index = index % length  # taking index within the proxy list size
    else:
        continue  # to reuse the same proxy

In this code, the for loop keeps on sending the requests with a single proxy. If a request returns a status code other than 200 (success code), it changes the index and uses the next proxy from the list. 

Conclusion

Any scraping task performs exceedingly better when combined with proxies. One of the simpler ways is integrating proxies with the aiohttp library in Python to extract data at a greater scale.

Please feel free to contact us via email or live chat if you need assistance or have any questions.

Please be aware that this is a third-party tool not owned or controlled by Oxylabs. Each third-party provider is responsible for its own software and services. Consequently, Oxylabs will have no liability or responsibility to you regarding those services. Please carefully review the third party's policies and practices and/or conduct due diligence before accessing or using third-party services.