Dive into code samples
Extracting data from complex websites has never been simpler. Discover the potential of our Web Scraper API with practical code samples.
Input parameters
source
Scraper
Set the scraper to 'google_search' to get web, image, or news search results. (Other sources: 'google', 'google_ads', 'google_images', 'google_lens', 'google_maps', 'google_travel_hotels', 'google_suggest', 'google_trends_explore')
query
Search query
Input UTF-encoded search query.
geo_location
Localization
Specify the UULE-based geographic location value. (State name, Country name, Coordinates and Radius, Criteria ID)
domain
Google domain
Specify Google domain you want to scrape.
locale
Interface language
Set the interface language.
results_language
Results language
Set the results language.
render
JavaScript rendering
Enable to load JavaScript-based content.
parse
Structured data
Enable to get structured data.
Input
Output
Copy
import requests
from pprint import pprint
username = "USERNAME"
password = "PASSWORD"
# Structure payload.
payload = {
"source": "google_search",
"query": "adidas",
"domain": "com",
"geo_location": "California,United States",
"parse": True
}
# Take a free trial or buy the product on our dashboard to create an API user.
# Replace 'USERNAME' and 'PASSWORD' with your API credentials to run this request.
# Get response by using real-time endpoint.
response = requests.request(
'POST',
'https://realtime.oxylabs.io/v1/queries',
auth=(username, password),
json=payload,
)
# Print prettified response to stdout.
pprint(response.json())See full code
"content": {
"url": "https://www.google.com/search?q=adidas&filter=1&safe=off&uule=w+CAIQICIYQ2FsaWZvcm5pYSxVbml0ZWQgU3RhdGVz&gl=us&hl=en",
"page": 1,
"results": {
"paid": [
{
"pos": 1,
"url": "https://www.adidas.com/us",
"desc": "Sign Up For adiClub To Get Free Standard Shipping And Start Earning Points On Every Order. Shop The adidas® Official Site. Join Our adiClub Loyalty Program. Sign Up & Get 15% Off. Gift Cards Available. View Size Charts. Free Shipping Available.",
"title": "adidas® Official Website | Place Your Order Today | adidas US",
"data_rw": "https://www.google.com/aclk?sa=l&ai=DChcSEwjJ_POllYCGAxW_VEgAHe29DH0YABAAGgJjZQ&ae=2&gclid=EAIaIQobChMIyfzzpZWAhgMVv1RIAB3tvQx9EAAYASAAEgKwcfD_BwE&sig=AOD64_2LcoUnvekUgtnHdwRH-W1AE5U96g&q&adurl",
"sitelinks": {
"expanded": [
{
"url": "https://www.adidas.com/us/men",
"title": "Men"
},
{
"url": "https://www.adidas.com/us/men",
"title": "Men's Collection"
}
]
},
"url_shown": "https://www.adidas.com",
"pos_overall": 1
}
],
"images": {
"items": [
{
"alt": "adidas Superstar Shoes - Black | Unisex Lifestyle | adidas US",
"pos": 1,
"url": "https://www.adidas.com/us/superstar-shoes/EG4959.html",
"data": "data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"source": "https://www.adidas.com/us/superstar-shoes/EG4959.html"
},
{
"alt": "adidas Superstar Shoes - White | Unisex Lifestyle | adidas US",
"pos": 2,
"url": "https://www.adidas.com/us/superstar-shoes/EG4958.html",
"data": "data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"source": "https://www.adidas.com/us/superstar-shoes/EG4958.html"
},
{
"alt": "Adidas Campus 00S Shoes - unisex - Better Scarlet Red / Clear Pink / Cloud White - M 14 / W 15",
"pos": 3,
"url": "https://www.adidas.com/us/campus-00s-shoes/H03477.html",
"data": "data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==",
"source": "https://www.adidas.com/us/campus-00s-shoes/H03477.html"
}
],
"pos_overall": 9
},
"organic": [
{
"pos": 1,
"url": "https://www.adidas.com/us",
"desc": "adidas is about more than sportswear and workout clothes. We partner with the best in the industry to co-create. This way we offer our fans the sporting goods, ...",
"title": "adidas Official Website | adidas US",
"sitelinks": {
"expanded": [
{
"url": "https://www.adidas.com/us/women",
"title": "Women's Shoes, Clothing and ..."
},
{
"url": "https://www.adidas.com/us/men-shoes",
"title": "Men's Shoes"
},
{
"url": "https://www.adidas.com/us/men",
"title": "Men's Shoes, Clothing and ..."
},
{
"url": "https://www.adidas.com/us/kids",
"title": "Kids' Shoes and Clothing"
},
{
"url": "https://www.adidas.com/us/originals",
"title": "Originals Sneakers & Clothing"
}
]
},
"url_shown": "https://www.adidas.com› ...",
"pos_overall": 2
},
...Successful scraping at scale with Oxylabs' Web Scraper API
Most popular search engines
City-level SERP data delivery from major search engines.
From $49/month
E-Commerce websites
E-commerce product page scraping with ready-to-use data.
From $49/month
Other websites
Scalable real-time data collection from a majority of websites.
Best for:
From $49/month
Simple integration
Step 1: Put your endpoint URL, API user credentials, and data payload into a single request.
Step 2: Send the request to our API. We'll take care of everything from there – you don't need to do anything else.
Step 3: Get the result either directly from the API or to your chosen cloud storage bucket.
Copy
import requests
from pprint import pprint
username = "USERNAME"
password = "PASSWORD"
payload = {
"source": "universal",
"url": "https://sandbox.oxylabs.io/products/",
"geo_location": "United States",
}
response = requests.request(
'POST',
'https://realtime.oxylabs.io/v1/queries',
auth=(username, password),
json=payload,
)
pprint(response.json())Headless Browser
Render JavaScript-based pages with a single line of code, eliminating the need for complex browser development or automated third-party tools. Set up custom browser instructions and enable Headless Browser to execute mouse clicks, input text, scroll pages, wait for elements to appear, and more.
Effortless JavaScript rendering
Browser instructions execution
Seamless data collection
Advanced features
Leverage Web Scraper API smart features for collecting real-time data.
Proxy management
ML-driven proxy selection and rotation using our premium proxy pool from 195 countries.
Custom parameters
Boost your scraping control with custom headers and cookies at no extra cost.
AI-driven fingerprinting
Unique HTTP headers, JavaScript, and browser fingerprints ensure resilience to dynamic content.
CAPTCHA bypass
Automatic retries and CAPTCHA bypassing for uninterrupted data retrieval.
Response recognition
ML-based recognition and adaptation to various responses, optimizing scraping efficiency.
Continuous scroll support
Set the amount of Google organic search results per SERP to suit your needs.
Scheduler
Automate recurring scraping jobs with desired frequency and receive data to AWS S3 or GCS.
Custom Parser
Define your own parsing logic using XPath or CSS selectors for structured data collection.
What do others say?
"Building and maintaining your own scraping and parsing solution is expensive. That’s why we turned to Oxylabs. They offered one of the best price-to-value combinations in the market and helped us save our total web scraping costs."
Wei Zheng
Chief Product Officer at Conductor
Oxylabs products fuel top industries with public web data
Harness public data for e-commerce success
Access valuable e-commerce data, such as pricing, product information, or reviews, on a large scale.
Price monitoring Dynamic pricing Monitor product information Identify MAP policy violations Learn more
Boost cybersecurity operations with public data
Gather critical data while staying unnoticed by threat actors to make proactive security decisions. Threat intelligence Email protection Prevent phishing attacks Learn more
Secure brands with top-tier public data collection solutions
Retrieve public data on a large scale to stop copyright infringement and counterfeiting. Copyright infringement monitoring Anti-counterfeit tracking Domain squatting detection Learn more
Unleash SEO potential with real-time public SERP data
Extract highly localized search engine data to monitor rankings and brand visibility.
Monitor SERP results Backlink intelligence Ad intelligence Learn more
Discover new opportunities with public travel data
Leverage real-time flight and hotel data to shape your travel business strategy with evidence-based insights.
Real-time ticket pricing data
Hotel pricing trends
Useful resources
Technical API documentation
Discover available scraping parameters and explore code examples for specific targets.
Oxylabs GitHub repositories
Learn how to scrape websites, use our tools, integrate products, and more.
Setup guide for Web Scraper API
Quickly integrate our Web Scraper API into your infrastructure.
Available scrapers
Frequently Asked Questions
What are web scraping services?
Oxylabs' web scraping services include an all-in-one web scraping solution tailored to suit a specific niche of data extraction, such as collecting information from e-commerce websites, search engines, and other targets.
The web services provide a complete infrastructure developed to simplify data extraction with generalized commands, automatic data structurization (parsing), simple data delivery, and a full range of auxiliary tools like Scheduler, Web Crawler, OxyCopilot, and more.
Which coding language is best for web scraping?
The best coding language for web scraping depends on many factors, including your specific requirements, existing skill set, and the complexity of the task at hand. Python is often considered the best language for web scraping due to its wide variety of dedicated libraries and ease of use, making it the top choice for beginners and professionals alike.
Is web scraping legal?
The legality of web scraping depends on the website's terms of service, the nature of the data, and your jurisdiction. Always review the website’s terms and consult with a legal professional to ensure compliance with relevant laws and regulations.
Get the latest news from data gathering world
Scale up your business with Oxylabs®