Back to blog

Continuous Scroll in Google Search: What It Is & How to Handle It in Web Scraping

Augustas Pelakauskas

2023-08-102 min read

When it comes to Google Search, users tend to avoid the second SERP (Search Engine Results Page). The (rather irrational) psychological factor is very strong, as visiting the second or subsequent page equals a loss of hope of finding relevant results.

The intent of the continuous scroll, a feature of Google SERPs, is to encourage searchers to look beyond the first few and scroll through more results. Compared to visiting the second SERP, scrolling downwards doesn’t appear nearly as futile.

The continuous scroll might benefit the second-page dwellers with more exposure without the need to improve their organic ranking. On the other hand, such a feature presents a roadblock in public web data extraction.

What is Google’s continuous scroll?

It’s a Google Search page navigation feature introduced in October of 2021. Primarily a mobile feature, the continuous scroll also made its way to desktops in December of 2022.

The traditional numbered page approach to displaying SERPs is discarded in favor of a scroll. To continue, you must select ''more results'' instead of a subsequent page. That’s why it’s a continuous and not an infinite scroll.

A challenge in web scraping

Dynamic page rendering, such as the continuous scroll layout in Google SERPs, is a challenge in web scraping. Results with such a layout contain only around the first 10 organic results in the initial HTTP response, as additional organic results are only loaded when a user scrolls down the page in their browser.

It presents a problem as paged layout would easily allow data extraction, for example, the first 100 organic results. With the continuous scroll, it’s no longer the case, and additional measures are needed to extract more than the initial results.

The capabilities of Web Scraper API

Handling of Google Search continuous scroll is a free feature of Oxylabs Web Scraper API. In some regions, the continuous scroll update is not yet available. Thus, when present, the API detects such layouts automatically and ensures that the user-requested amount of organic results is loaded and scraped.

What is Web Scraper API?

Web Scraper API is a tool for real-time, large-volume public web data extraction. With no geographic location or bandwidth limitations, you can extract data from localized SERPs in 195 countries and territories with coordinate-level precision.

Packed with features such as Custom Parser and Scheduler, it’s an all-in-one solution for maintenance-free web scraping.

With Web Scraper API, you can specify how many organic results per SERP you would like to scrape. You can do so with a custom request using a desired parameter value.

The API requests Google for additional results until it gathers the amount specified in your query. The final results, HTML or parsed data in JSON format, are then delivered to you. Around half of the web scraping tasks performed by our clients have a continuous scroll layout.

Take a look at our documentation for exact details on setting up queries to suit your specific web scraping case.

Summary

You can try Web Scraper API with 5K results for a week for free and see how it handles results from Google Search with the continuous scroll.

If you would like to know more about our web scraping solutions or have any questions about handling the continuous scroll, please contact us via the 24/7 live chat on our homepage or email us at support@oxylabs.io.

We advise you to seek legal consultation before engaging in any kind of scraping activities in order to assess a specific situation and get an expert opinion on further proceedings.

About the author

Augustas Pelakauskas

Senior Copywriter

Augustas Pelakauskas is a Senior Copywriter at Oxylabs. Coming from an artistic background, he is deeply invested in various creative ventures - the most recent one being writing. After testing his abilities in the field of freelance journalism, he transitioned to tech content creation. When at ease, he enjoys sunny outdoors and active recreation. As it turns out, his bicycle is his fourth best friend.

Learn more about Augustas Pelakauskas

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.