The demand for digital content has increased exponentially. Since the resulting competition increases, the existing websites are rapidly changing and updating their structure.
Quick updates are beneficial to general consumers. However, it’s a considerable hassle for a specific portion of businesses that perform public data collection since web scraping uses routines tailored for specific conditions of the individual websites, and the frequent updates tend to disrupt them. This is where RegEx comes into play by alleviating some of the more complex elements of certain acquisition and parsing processes.
RegEx stands for Regular Expressions, a method to match specific patterns depending on the provided combinations, which can be used as filters to get the desired output.
RegEx can be used to validate all types of character combinations, including special characters like line breaks. One of the biggest pros Regular Expressions have is that no matter the type of data/input (irrespective of its size), it’s always compared to the same single regular expression, making the code more efficient.
Regular Expressions are universal and can be implemented in any programming language.
|^||Start of a string|
|$||End of a string|
|.||Any character (except \n)|
||||Characters on either side of the symbol|
|\||Escapes special characters|
|Char||The character given|
|*||Any number of previous characters|
|?||1 previous character|
|+||1 or more previous characters|
|\s||Any whitespace character|
|\w||Any word character|
|\b||Word boundary character|
|\D||Inverse of \d|
|\S||Inverse of \s|
|\W||Inverse of \w|
In this tutorial, the RegEx scraping target is book titles and prices from a dummy website intended for training purposes.
The latest version of Python.
Beautiful Soup 4 library to parse HTML.
Requests library to make HTML requests.
Let’s begin with creating a virtual environment for the project:
python3 -m venv scrapingdemo
Activate the newly created virtual environment (the example for Linux):
Now, install the required Python modules.
Requests is a library responsible for sending requests to the websites on the internet and returning their response. To install Requests, enter the following:
pip install requests
Beautiful Soup is a module used to parse and extract data from the HTML response. To install Beautiful Soup, enter the following:
pip install beautifulsoup4
re is a built-in Python module responsible for working with Regular Expressions.
Next, create an empty Python file, for example, demo.py.
To import the required libraries, enter the following:
import requests from bs4 import BeautifulSoup import re
Use the Requests library to send a request to a web page from which you want to scrape the data. In this case, https://books.toscrape.com/. To commence, enter the following:
page = requests.get('https://books.toscrape.com/')
First, create a Beautiful Soup object and pass the page content received from your request during the initialization, including the parser type. As you’re working with an HTML code, select HTML.parser as the parser type.
Inspecting the HTML code element
By inspecting the elements (right-click and select inspect element) in a browser, you can see that each book title and price are presented inside an article element with the class called product_pod. Use Beautiful Soup to get all the data inside these elements, and then convert it to a string:
soup = BeautifulSoup(page.content, 'html.parser') content = soup.find_all(class_='product_pod') content = str(content)
Since the acquired content has a lot of unnecessary data, create two regular expressions to get only the desired data.
Content of the acquired data
Finding the pattern
First, inspect the title of the book to find the pattern. You can see above that every title is present after the text title= in the format title=“Titlename”.
Generating the expression
Then, create an expression that returns the data inside quotations after the title= by specifying "(.*?)".
The first expression is as follows:
re_titles = r'title="(.*?)">'
Finding the pattern
First, inspect the price of the book. Every price is present after the text £ in the format £=price before the paragraph tag </p>.
Generating the expression
Then, create an expression that returns the data inside quotations after the £= and before the </p> by specifying £(.*?)</p>.
The second expression is as follows:
re_prices = '£(.*?)</p>'
To conclude, use the expressions with re.findall to find the substrings matching the patterns. Lastly, save them in the variables title_list and price_list.
titles_list = re.findall(re_titles, content) price_list = re.findall(re_prices, content)
To save the output, loop over the pairs for the titles and prices and write them to the output.txt file.
with open("output.txt", "w") as f: for title, price in zip(titles_list, price_list): f.write(title + "\t" + price + "\n")
The output file
Putting everything together, this is the complete code that can be run by calling python demo.py:
# Importing the required libraries. import requests from bs4 import BeautifulSoup import re # Requesting the HTML from the web page. page = requests.get("https://books.toscrape.com/") # Selecting the data. soup = BeautifulSoup(page.content, "html.parser") content = soup.find_all(class_="product_pod") content = str(content) # Processing the data using Regular Expressions. re_titles = r'title="(.*?)">' titles_list = re.findall(re_titles, content) re_prices = "£(.*?)</p>" price_list = re.findall(re_prices, content) # Saving the output. with open("output.txt", "w") as f: for title, price in zip(titles_list, price_list): f.write(title + "\t" + price + "\n")
This article explained what Regular Expressions are, how to use them, and what most commonly used tokens do. An example of scraping the titles and prices from a web page utilizing Python and Regular Expressions was also provided. If you're looking for an advanced web scraping solution, feel free to explore the features of our Web Scraper API.
About the author
Augustas Pelakauskas is a Senior Copywriter at Oxylabs. Coming from an artistic background, he is deeply invested in various creative ventures - the most recent one being writing. After testing his abilities in the field of freelance journalism, he transitioned to tech content creation. When at ease, he enjoys sunny outdoors and active recreation. As it turns out, his bicycle is his fourth best friend.
All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.
Get the latest news from data gathering world
Forget about complex web scraping processes
Choose Oxylabs' advanced web intelligence collection solutions to gather real-time public data hassle-free.
Scale up your business with Oxylabs®
GET IN TOUCH
Certified data centers and upstream providers
Connect with us
Advanced proxy solutions