Proxy locations

Europe

North America

South America

Asia

Africa

Oceania

See all locations

Network statusCareers

Home

products

Scraper APIs

Web Scraping Services From $49/Month

Scalable real-time data collection from search engines, e-commerce marketplaces, and any other target of your choice.

See Web Scraper API in action

Success-based payment

24/7 support

CAPTCHA bypass

JavaScript rendering

Web Scraper API

Dive into code samples

Extracting data from complex websites has never been simpler. Discover the potential of our Web Scraper API with practical code samples.

Input parameters

source

Scraper

Set the scraper to 'google_search' to get web, image, or news search results. (Other sources: 'google', 'google_ads', 'google_images', 'google_lens', 'google_maps', 'google_travel_hotels', 'google_suggest', 'google_trends_explore')

query

Search query

Input UTF-encoded search query.

geo_location

Localization

Specify the UULE-based geographic location value. (State name, Country name, Coordinates and Radius, Criteria ID)

domain

Google domain

Specify Google domain you want to scrape.

locale

Interface language

Set the interface language.

results_language

Results language

Set the results language.

render

JavaScript rendering

Enable to load JavaScript-based content.

parse

Structured data

Enable to get structured data.

Input

Output

Output preview

Copy

import requests
from pprint import pprint
username = "USERNAME"
password = "PASSWORD"
# Structure payload.
payload = {
    "source": "google_search",
    "query": "adidas",
    "domain": "com",
    "geo_location": "California,United States",
    "parse": True
}
# Take a free trial or buy the product on our dashboard to create an API user.
# Replace 'USERNAME' and 'PASSWORD' with your API credentials to run this request.
# Get response by using real-time endpoint.
response = requests.request(
    'POST',
'https://realtime.oxylabs.io/v1/queries',
    auth=(username, password),
    json=payload,
)
# Print prettified response to stdout.
pprint(response.json())

See full code

"content": {
    "url": "https://www.google.com/search?q=adidas&filter=1&safe=off&uule=w+CAIQICIYQ2FsaWZvcm5pYSxVbml0ZWQgU3RhdGVz&gl=us&hl=en",
    "page": 1,
    "results": {
        "paid": [
            {
                "pos": 1,
                "url": "https://www.adidas.com/us",
                "desc": "Sign Up For adiClub To Get Free Standard Shipping And Start Earning Points On Every Order. Shop The adidas® Official Site. Join Our adiClub Loyalty Program. Sign Up & Get 15% Off. Gift Cards Available. View Size Charts. Free Shipping Available.",
                "title": "adidas® Official Website | Place Your Order Today | adidas US",
                "data_rw": "https://www.google.com/aclk?sa=l&ai=DChcSEwjJ_POllYCGAxW_VEgAHe29DH0YABAAGgJjZQ&ae=2&gclid=EAIaIQobChMIyfzzpZWAhgMVv1RIAB3tvQx9EAAYASAAEgKwcfD_BwE&sig=AOD64_2LcoUnvekUgtnHdwRH-W1AE5U96g&q&adurl",
                "sitelinks": {
                    "expanded": [
                        {
                            "url": "https://www.adidas.com/us/men",
                            "title": "Men"
                        },
                        {
                            "url": "https://www.adidas.com/us/men",
                            "title": "Men's Collection"
                        }
                    ]
                },
                "url_shown": "https://www.adidas.com",
                "pos_overall": 1
            }
        ],
        "images": {
            "items": [
                {
                    "alt": "adidas Superstar Shoes - Black | Unisex Lifestyle | adidas US",
                    "pos": 1,
                    "url": "https://www.adidas.com/us/superstar-shoes/EG4959.html",
                    "data": "data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
                    "source": "https://www.adidas.com/us/superstar-shoes/EG4959.html"
                },
                {
                    "alt": "adidas Superstar Shoes - White | Unisex Lifestyle | adidas US",
                    "pos": 2,
                    "url": "https://www.adidas.com/us/superstar-shoes/EG4958.html",
                    "data": "data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
                    "source": "https://www.adidas.com/us/superstar-shoes/EG4958.html"
                },
                {
                    "alt": "Adidas Campus 00S Shoes - unisex - Better Scarlet Red / Clear Pink / Cloud White - M 14 / W 15",
                    "pos": 3,
                    "url": "https://www.adidas.com/us/campus-00s-shoes/H03477.html",
                    "data": "data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
                    "source": "https://www.adidas.com/us/campus-00s-shoes/H03477.html"
                }
            ],
            "pos_overall": 9
        },
        "organic": [
            {
                "pos": 1,
                "url": "https://www.adidas.com/us",
                "desc": "adidas is about more than sportswear and workout clothes. We partner with the best in the industry to co-create. This way we offer our fans the sporting goods, ...",
                "title": "adidas Official Website | adidas US",
                "sitelinks": {
                    "expanded": [
                        {
                            "url": "https://www.adidas.com/us/women",
                            "title": "Women's Shoes, Clothing and ..."
                        },
                        {
                            "url": "https://www.adidas.com/us/men-shoes",
                            "title": "Men's Shoes"
                        },
                        {
                            "url": "https://www.adidas.com/us/men",
                            "title": "Men's Shoes, Clothing and ..."
                        },
                        {
                            "url": "https://www.adidas.com/us/kids",
                            "title": "Kids' Shoes and Clothing"
                        },
                        {
                            "url": "https://www.adidas.com/us/originals",
                            "title": "Originals Sneakers & Clothing"
                        }
                    ]
                },
                "url_shown": "https://www.adidas.com› ...",
                "pos_overall": 2
            },
...

Try Web Scraper API out for yourself

Discover Web Scraper API Playground on the dashboard for a firsthand interaction with our solution, and explore technical documentation for all the information you need.

Successful scraping at scale with Oxylabs' Web Scraper API

Most popular search engines

City-level SERP data delivery from major search engines.

Best for:

Google, Bing, Baidu, Yandex

E-Commerce websites

E-commerce product page scraping with ready-to-use data.

Other websites

Scalable real-time data collection from a majority of websites.

Simple integration

Step 1: Put your endpoint URL, API user credentials, and data payload into a single request.

Step 2: Send the request to our API. We'll take care of everything from there – you don't need to do anything else.

Step 3: Get the result either directly from the API or to your chosen cloud storage bucket.

See more examples

Copy

import requests
from pprint import pprint

username = "USERNAME"
password = "PASSWORD"

payload = {
    "source": "universal",
    "url": "https://sandbox.oxylabs.io/products/",
    "geo_location": "United States",
}

response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=(username, password),
    json=payload,
)

pprint(response.json())

Headless Browser

Render JavaScript-based pages with a single line of code, eliminating the need for complex browser development or automated third-party tools. Set up custom browser instructions and enable Headless Browser to execute mouse clicks, input text, scroll pages, wait for elements to appear, and more.

  • Effortless JavaScript rendering

  • Browser instructions execution

  • Seamless data collection

Learn more

Advanced features

Leverage Web Scraper API smart features for collecting real-time data.

Proxy management

ML-driven proxy selection and rotation using our premium proxy pool from 195 countries.

Custom parameters

Boost your scraping control with custom headers and cookies at no extra cost.

AI-driven fingerprinting

Unique HTTP headers, JavaScript, and browser fingerprints ensure resilience to dynamic content.

CAPTCHA bypass

Automatic retries and CAPTCHA bypassing for uninterrupted data retrieval.

Response recognition

ML-based recognition and adaptation to various responses, optimizing scraping efficiency.

Continuous scroll support

Set the amount of Google organic search results per SERP to suit your needs.

Scheduler

Automate recurring scraping jobs with desired frequency and receive data to AWS S3 or GCS.

Custom Parser

Define your own parsing logic using XPath or CSS selectors for structured data collection.

What do others say?

"Building and maintaining your own scraping and parsing solution is expensive. That’s why we turned to Oxylabs. They offered one of the best price-to-value combinations in the market and helped us save our total web scraping costs."

Wei Zheng

Chief Product Officer at Conductor

Read full story
More customer stories

Oxylabs products fuel top industries with public web data

Harness public data for e-commerce success

Access valuable e-commerce data, such as pricing, product information, or reviews, on a large scale.

Boost cybersecurity operations with public data

Gather critical data while staying unnoticed by threat actors to make proactive security decisions.

 

Secure brands with top-tier public data collection solutions

Retrieve public data on a large scale to stop copyright infringement and counterfeiting.

Unleash SEO potential with real-time public SERP data

Extract highly localized search engine data to monitor rankings and brand visibility.

 

Discover new opportunities with public travel data

Leverage real-time flight and hotel data to shape your travel business strategy with evidence-based insights.

  • Real-time ticket pricing data

  • Hotel pricing trends

Frequently Asked Questions

What are web scraping services?

Oxylabs' web scraping services include an all-in-one web scraping solution tailored to suit a specific niche of data extraction, such as collecting information from e-commerce websites, search engines, and other targets.

The web services provide a complete infrastructure developed to simplify data extraction with generalized commands, automatic data structurization (parsing), simple data delivery, and a full range of auxiliary tools like Scheduler, Web Crawler, OxyCopilot, and more.

Which coding language is best for web scraping?

The best coding language for web scraping depends on many factors, including your specific requirements, existing skill set, and the complexity of the task at hand. Python is often considered the best language for web scraping due to its wide variety of dedicated libraries and ease of use, making it the top choice for beginners and professionals alike.

Is web scraping legal?

The legality of web scraping depends on the website's terms of service, the nature of the data, and your jurisdiction. Always review the website’s terms and consult with a legal professional to ensure compliance with relevant laws and regulations.

Get the latest news from data gathering world

I'm interested