Asynchronous Web Scraping With Python & AIOHTTP

What is asynchronous web scraping? 

Asynchronous web scraping, also referred to as non-blocking or concurrent, is a special technique that allows you to begin a potentially lengthy task and still have a chance to respond to other events, rather than having to wait for that long task to finish.

Additionally, with asyncio library providing a great number of tools to write the non-blocking code and AIOHTTP delivering even more precise functionality for HTTP requests, it is no surprise that asynchronous scraping gained so much popularity among developers. 

Remember that even though this blog post demonstrates the asynchronous approach's benefits, scraping multiple URLs at once can also be achieved through multiple threads and processes.. For more details, check out one of our other step-by-step tutorials. 

Sync vs. async: What’s the difference?

Compared to async, synchronous approach to scraping multiple website URLs refers to the activity of running one request at a time. In essence, when using this code, a user will process one URL and move on to processing the next URL only after the previous one has completely finished processing. 

Thus, the main difference between the two approaches is that synchronous code stops the next request from running while async code allows you to scrape data from multiple web pages roughly at the same time. This leads us to the main benefit of the asynchronous approach – great time-efficiency. You no longer have to wait for the scraping of one page to finish before starting the other. 

Now that you have a clear understanding of the main difference between sync and async code, we can move on to the web scraping tutorial part and create a sample Python code for each of the approaches.

Sending asynchronous HTTP requests

Let’s take a look at the asynchronous Python tutorial. For this use-case, we will use the aiohttp module.

1. Create an empty python file with a main function

Note that the main function is marked as asynchronous. We use asyncio loop to prevent the script from exiting until the main function completes.

import asyncio


async def main():
    print('Saving the output of extracted information')


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Once again, it is a good idea to track the performance of your script. For that purpose, let's write a code that tracks script execution time.

2. Track script execution time

As with the first example, record the time at the start of the script. Then, type in any code that you need to measure (currently a single print statement). Finally, calculate how much time has passed by taking the current time and subtracting the time at the start of the script. Once we have how much time has passed, we print it while rounding the resulting float to the last 2 decimals.

import asyncio
import time


async def main():
    start_time = time.time()

    print('Saving the output of extracted information')

    time_difference = time.time() - start_time
    print(f'Scraping time: %.2f seconds.' % time_difference)


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Time to read the csv file that contains URLs. The file will contain a single column called url. There, you will see all the URLs that need to be scraped for data.

Sending synchronous HTTP requests

In this tutorial we are going to scrape URLs defined in urls.csv using a synchronous approach. For this particular use case, the python requests module is an ideal tool. 

1. Create a Python file with a main function

def main():
    print('Saving the output of extracted information')

main()

Tracking the performance of your script is always a good idea. Therefore, the next step is to add a code that tracks script execution time. 

2. Track script execution time

First, record time at the very start of the script. Then, type in any code that needs to be measured – in this case, we are using a single print statement. Finally, calculate how much time has passed. This can be done by taking the current time and subtracting the time at the start of the script. Once we know how much time has passed, we can print it while rounding the resulting float to the last 2 decimals.

import time


def main():
    start_time = time.time()

    print('Saving the output of extracted information')

    time_difference = time.time() - start_time
    print(f'Scraping time: %.2f seconds.' % time_difference)

main()

Now that the preparations are done, it's time to read the csv file that contains URLs. There, you will see a single column called url, which will contain URLs that have to be scraped for data.

3. Create a loop

Next, we have to open up urls.csv. After that, load it using the csv module and loop over each and every URL from the csv file.

import csv
import time


def main():
    start_time = time.time()

    print('Saving the output of extracted information')
    with open('urls.csv') as file:
        csv_reader = csv.DictReader(file)
        for csv_row in csv_reader:
            # the url from csv can be found in csv_row['url']
            print(csv_row['url'])

    time_difference = time.time() - start_time
    print(f'Scraping time: %.2f seconds.' % time_difference)

main()

At this point, the job is almost done - all that’s left to do is to scrape it, although before you do that, look at the data you’re scraping.

The title of the book “A Light in the Attic” can be extracted from an <h1> tag, that is wrapped by a <div> tag with a product_main class.

Comparing the performance of sync and async

Now that we carefully went through the processes of making requests with both synchronous and asynchronous methods, we can run the requests once again and compare the performance of two scripts.

The time difference is huge – while the async web scraping code was able to execute all the tasks in around 3 seconds, it took almost 16 for the synchronous one. This proves that scraping asynchronously is indeed more beneficial due to its noticeable time efficiency.

Wrapping up

Collection of high-quality public data often requires the scraping of dozens or even hundreds of web pages. To help you get an understanding of how this can be done efficiently, we compared two different approaches of scraping multiple website URLs with Python – synchronous and asynchronous (non-blocking). By exploring their main difference and providing sample code snippets, we demonstrated that asynchronous http requests method is more advantageous for sending multiple requests. 

If you have any questions about this particular tutorial or any other topic related to web scraping, feel free to contact us at hello@oxylabs.io. Additionally, take advantage of a free trial of our web intelligence solutions, such as Web Scraper API. Try them out for 1-week and decide whether they fit your data needs.