Web Scraping With PHP

PHP is a general-purpose scripting language and one of the most popular options for web development. For example, WordPress, the most common content management system to create websites, is built using PHP.

PHP offers various building blocks required to build a web scraper, although it can quickly become an increasingly complicated task. Conveniently, there are many open-source libraries that can make web scraping with PHP more accessible.

This article will guide you through the step-by-step process of writing various web scraping PHP routines that can extract public data from static and dynamic web pages.

For your convenience, we also prepared this tutorial in a video format:

Upon examining the HTML of the target web page, we can see that each book is contained in an article tag, which has a product_pod class. Here, the CSS Selector would be .product_pod.

In each article tag, the complete book title is located in the thumbnail image as an alt attribute value. The CSS Selector for the book title would be .image_container img.

Finally, the CSS Selector for the book price would be .price_color.

To get all of the titles and prices from this page, first, we need to locate the container and then run the each loop.

In this loop, an anonymous function will extract and print the title along with the price as follows:

function scrapePage($url, $client){
    $crawler = $client->request('GET', $url);
    $crawler->filter('.product_pod')->each(function ($node) {
            $title = $node->filter('.image_container img')->attr('alt');
            $price = $node->filter('.price_color')->text();
            echo $title . "-" . $price . PHP_EOL;
        });
    }

The functionality of web data extraction was isolated in a function. The same function can be used for extracting data from different websites.

Handling pagination

At this point, our PHP web scraper is performing data extraction from only a single URL. In real-life web scraping scenarios, multiple pages would be involved.

In this particular site, the pagination is controlled by a Next link (button). The CSS Selector for the Next link is .next > a.

In the function scrapePage that we’ve created earlier, add the following lines:

try {
    $next_page = $crawler->filter('.next > a')->attr('href');
} catch (InvalidArgumentException) { //Next page not found
    return null;
}
return "https://books.toscrape.com/catalogue/" . $next_page;

This code uses the CSS Selector to locate the Next button and to extract the value of the href attribute, returning the relative URL of the subsequent page. On the last page, this line of code will raise the InvalidArgumentException.

If the next page is found, this function will return its URL. Otherwise, it will return null.

From now on, we’ll be initiating each scraping cycle with a different URL. This will make the conversion from a relative URL to an absolute one easier.

Lastly, we can use a while loop to call this function:

$client = new Client();
$nextUrl = "https://books.toscrape.com/catalogue/page-1.html";
while ($nextUrl) {
    $nextUrl = scrapePage($nextUrl, $client);
}
scrapePage($url, $client);

The web scraping code is almost complete.

Writing data to a CSV file

The final step of the web scraping PHP process is to export the data to a storage. PHP’s built-in fputcsv function can be used to export the data to a CSV file.

First, open the CSV file in write or append mode and store the file handle in a variable.

Next, send the variable to the scrapePage function. Then, call the fputcsv function for each book to write the title and price in one row.

Lastly, after the while loop, close the file by calling fclose.

The final code file will be as follows:

function scrapePage($url, $client, $file)
{
    $crawler = $client->request('GET', $url);
    $crawler->filter('.product_pod')->each(function ($node) use ($file) {
        $title = $node->filter('.image_container img')->attr('alt');
        $price = $node->filter('.price_color')->text();
        fputcsv($file, [$title, $price]);
    });
    try {
        $next_page = $crawler->filter('.next > a')->attr('href');
    } catch (InvalidArgumentException) { //Next page not found
        return null;
    }
    return "https://books.toscrape.com/catalogue/" . $next_page;
}

$client = new Client();
$file = fopen("books.csv", "a");
$nextUrl = "https://books.toscrape.com/catalogue/page-1.html";

while ($nextUrl) {
    echo "<h2>" . $nextUrl . "</h2>" . PHP_EOL;
    $nextUrl = scrapePage($nextUrl, $client, $file);
}
fclose($file);

Run this file from the terminal:

php books.php

This will create a books.csv file with 1,000 rows of data.

Web scraping with Guzzle, XML, and XPath

Guzzle is a PHP library that sends HTTP requests to web pages in order to get a response. In other words, Guzzle is a PHP HTTP client that you can use to scrape data. Do note that before we can work with a web page, we need to understand two more concepts--XML and XPath.

XML stands for eXtensible Markup Language. It will be used to create files for storing structured data. These files can then be transmitted, and the data constructed.

There is the issue of reading  XML files and this is where XPath comes into the picture.

XPath stands for XML Path and is used for navigation and selecting XML nodes.

HTML files are very similar to XML files. In some cases we might need a parser to make adjustments to the minor differences and make HTML at least somewhat compliant with XML file standards. Crucially there are some parsers that can read even poorly formatted XML. 

In any case, the parsers will then make necessary HTML modifications so that we can work with XPath to query and navigate the HTML.

Setting up a Guzzle Project

To install Guzzle, create a directory where you intend to keep the source code. Navigate to the directory and enter these commands:

composer init --no-interaction --require="php >=7.1"
composer require guzzlehttp/guzzle

In addition to Guzzle, we’ll also use a library for parsing HTML code. There are many PHP libraries available such as simple HTML dom parser and Symphony DOMCrawler. In this tutorial, Symphony DOMCrawler is chosen as the syntax is very similar to Goutte and you will be able to apply what you already know in this section. Another point in favor of DomCrawler over the simple HTML dom parser is that it supports working with invalid HTML code very well.

Install DOMCrawler using the following command:

composer require symfony/dom-crawler

These commands will download all the necessary files. The next step is to create a new file and save it as scraper.php.

Sending HTTP requests with Guzzle

Similar to Goutte, the most important class of Guzzle is Client. Begin by creating a new file scraper.php and enter the following lines of PHP code:

<?php
require 'vendor/autoload.php';
use GuzzleHttp\Client;
use Symfony\Component\DomCrawler\Crawler;

Now we are ready to create an object of the Client class:

$client = new Client();

The client object can then be used to send a request. The method to send the request is conveniently called request. It takes two parameters — the HTTP method and the target URL, and returns a response:

$response = $client->request('GET', 'https://books.toscrape.com');

From this response, we can extract the web page's HTML as follows:

$html = $response->getBody()->getContents();
echo $html

Note that in this example, the response contains HTML code. If you are working with a web page that returns JSON, you can save the JSON to a file and stop the script. The next section will be applicable only if the response contains HTML or XML data.

Continuing, we will use the DomCrawler to extract specific elements from this web page.

Locating HTML elements via XPath

Import the Crawler class and create an instance of the Crawler class as shown in the following PHP code snippet:

use Symfony\Component\DomCrawler\Crawler;

We can create an instance of the crawler class as follows:

$crawler = new Crawler($html);

Now we can use the filterXPath method to extract any XML node. For example, the following line prints only the title of the page:

echo $crawler->filterXPath('//title')->text();

A quick note about XML Nodes: In XML, everything is a node-- an element is a node, an attribute is a node, and text is also a node. The filterXPath method returns a node. Therefore, to extract the text from an element, even if you use the text() function in XPath, you still have to call the text() method to extract text as a string.

In other words, both the following lines of code will return the same value:

echo $crawler->filterXPath('//title')->text();
echo $crawler->filterXPath('//title/text()')->text();

Now, let's move on to extracting the book titles and prices.

Extracting the elements

Before we write web scraping code, let’s start with analyzing the HTML of our page.

Open the web page https://books.toscrape.com in Chrome, right-click on a book and select Inspect. 

The books are located in <article> elements with the class attribute set to product_pod. The XPath to select these nodes will be as follows:

//*[@class="product_pod"]

In each article tag, the complete book title is located in the thumbnail image as an alt attribute value. The XPath for book title and book price would be as follows:

//*[@class="image_container"]/a/img/@alt
//*[@class="price_color"]/text()

To get all of the titles and prices from this page, we first need to locate the container and then use a loop to get to each of the elements containing the data we need.

In this loop, an anonymous function will extract and print the title along with the price, as shown in the following PHP code snippet:

$crawler->filterXpath('//*[@class="product_pod"]')->each(function ($node) {
$title = $node->filterXpath('.//*[@class="image_container"]/a/img/@alt')->text();
$price = $node->filterXPath('.//*[@class="price_color"]/text()')->text();
echo $title . "-" . $price . PHP_EOL;
});

This was a simple demonstration of how you can scrape data from any page using Guzzle or DOMCrawler parsers. Note that this method will not work with a dynamic website. These websites use JavaScript code that cannot be handled by DOMCrawler. In cases like this, you will need to use Symphony Panther.

The next step after extracting data is to save it.

Saving extracted data to a file

To store the extracted data, you can change the script to use the built-in PHP and create a CSV file. 

Write the following PHP code snippet as this:

$file = fopen("books.csv", "a");
$crawler->filterXpath('//*[@class="product_pod"]')->each(function ($node) use ($file) {
$title = $node->filterXpath('.//*[@class="image_container"]/a/img/@alt')->text();
$price = $node->filterXPath('.//*[@class="price_color"]/text()')->text();
fputcsv($file, [$title, $price]);
});
fclose($file);

This code snippet, when run, will save all the data to the books.csv file.

Web scraping with Symfony Panther

Dynamic websites use JavaScript to render the contents. For such websites, Goutte wouldn’t be a suitable option.

For these websites, the solution is to employ a browser to render the page. It can be done using another component from Symfony – Panther. Panther is a standalone PHP library for web scraping using real browsers.

In this section, we’ll scrape quotes and authors from quotes.toscrape.com. It is a dummy website for learning the basics of scraping dynamic web pages.

Installing Panther and its dependencies

To install Panther, open the terminal, navigate to the directory where you will be storing your source code, and run the following commands:

composer init --no-interaction --require="php >=7.1" 
composer require symfony/panther
composer update

These commands will create a new composer.json file and install Symfony/Panther.

The other two dependencies are a browser and a driver. The common browser choices are Chrome and Firefox. The chances are that you already have one of these browsers installed.

The driver for your browser can be downloaded using any of the package managers.

On Windows, run:

choco install chromedriver

On macOS, run:

brew install chromedriver

Sending HTTP requests with Panther

Panther uses the Client class to expose the get() method. This method can be used to load URLs, or in other words, to send HTTP requests.

The first step is to create the Chrome Client. Create a new PHP file and enter the following lines of code:

<?php
require 'vendor/autoload.php';
use \Symfony\Component\Panther\Client;
$client = Client::createChromeClient();

The $client object can then be used to load the web page:

$client->get('https://quotes.toscrape.com/js/');

This line will load the page in a headless Chrome browser.

Locating HTML elements via CSS Selectors

To locate the elements, first, we need to get a reference for the crawler object. The best way to get an object is to wait for a specific element on a page using the waitFor() method. It takes the CSS Selector as a parameter:

$crawler = $client->waitFor('.quote');

The code line waits for the element with this selector to become available and then returns an instance of the crawler.

The rest of the code is similar to Goutte’s as both use the same CssSelector component of Symfony.

First, the filter method is supplied by the CSS Selector to get all of the quote elements. Then, the anonymous function is supplied to each quote to extract the author and the text:

    $crawler->filter('.quote')->each(function ($node) {
        $author = $node->filter('.author')->text();
        $quote = $node->filter('.text')->text();
       echo $autor." - ".$quote
    });

Handling pagination

To scrape data from all of the subsequent pages of this website, we can simply click the Next button. For clicking the links, the clickLink() method can be used. This method works directly with the link text.

On the last page, the link won’t be present, and calling this method will throw an exception. This can be handled by using a try-catch block:

while (true) {
    $crawler = $client->waitFor('.quote');
…
    try {
        $client->clickLink('Next');
    } catch (Exception) {
        break;
    }
}

Writing data to a CSV file

Writing the data to CSV is straightforward when using PHP’s fputcsv() function. Open the CSV file before the while loop, write every row using the fputcsv() function, and close the file after the loop.

Putting everything together, here is the final code:

$file = fopen("quotes.csv", "a");
while (true) {
    $crawler = $client->waitFor('.quote');
    $crawler->filter('.quote')->each(function ($node) use ($file) {
        $author = $node->filter('.author')->text();
        $quote = $node->filter('.text')->text();
        fputcsv($file, [$author, $quote]);
    });
    try {
        $client->clickLink('Next');
    } catch (Exception) {
        break;
    }
}
fclose($file);

Once you execute the web scraper contained in this PHP script, you will have a quotes.csv file with all the quotes and authors ready for further analysis.

Click here and check out a repository on GitHub to find the complete code used in this article.

Conclusion

You shouldn’t run into major hiccups when using Goutte for most static web pages, as this popular library offers sufficient functionality and extensive documentation. However, if the typical HTML extraction methods aren’t up to the task when dynamic elements come into play, then Symfony Panther is the right solution to deal with more complicated loads.

If you are working with a site developed using Laravel, Code Igniter, or just plain PHP, writing a web scraping part directly in PHP can be very useful, for example, when creating your own WordPress plugin. As PHP is also a scripting language, you can write web scraping code even when it is not meant to be deployed to a website.

If you want to know more on how to scrape the web using other programming languages, check similar articles, such as web scraping with JavaScript, Java, R, Ruby, Golang and Python on our blog. Additionally, use an opportunity to try the functionality of our own general-purpose web scraper for free.