Yahoo! Search Results Data

Vytenis Kaubrė

Last updated on

2025-03-12

3 min read

OxyCopilot makes it easy to generate scraping and parsing instructions for collecting public Yahoo! data - no coding required. Simply describe your requirements in plain English, and the system will automatically generate optimized parsing logic for use with Web Scraper API.

We've put together an example that demonstrates how to use OxyCopilot along with the key details needed to collect public search results data from Yahoo!.

Before using OxyCopilot, you need to register for our dashboard, claim a free trial (or choose a suitable plan) for our Web Scraper API, and create an API user. OxyCopilot lets you build scrapers while leveraging Oxylabs’ advanced scraping infrastructure, so you won’t be able to generate scraping and parsing instructions without the mentioned steps. You can find more information about it in the Getting Started part.

Summary

Input prompt

Copy

Parse the following sections from a search results page:
- Organic result listings:
  -- Title: Extract text only from the <a> element.
  -- Description.
  -- Link.
- Top Stories. From each card in the list, extract the following:
  -- Article title.
  -- Link.
  -- News source.
- People Also Ask: Parse the question, answer, and link of each FAQ.
- Related Searches list: Extract the values from the "title" attributes.

We make sure to keep this tutorial up-to-date, but errors may still occur. Feel free to regenerate the prompt or contact us if you need help.

An overview of OxyCopilot after providing all the necessary details to get results

Output

After completing all the steps and following our recommendations for using OxyCopilot, this is the final result you should expect:

Copy

{
    "top_stories": [
        {
            "link": "https://www.proactiveinvestors.com/companies/news/1068066/nvidia-gtc-conference-a-wake-up-call-for-tech-bulls-says-investment-bank-1068066.html",
            "news_source": "Proactive Investors",
            "article_title": "Nvidia GTC Conference a 'wake-up call' for tech bulls, says investment bank"
        },
        {
            "link": "https://finance.yahoo.com/news/why-nvidia-stock-holds-the-key-to-unlocking-a-rejuvenated-market-123646743.html?fr=sycsrp_catchall",
            "news_source": "Yahoo Finance",
            "article_title": "Why Nvidia stock holds the key to unlocking a rejuvenated market"
        },
        {
            "link": "https://finance.yahoo.com/news/where-nvidia-1-113000389.html?fr=sycsrp_catchall",
            "news_source": "Motley Fool via Yahoo Finance",
            "article_title": "Where Will Nvidia Be in 1 Year?"
        },
        {
            "link": "https://www.marketwatch.com/livecoverage/nvidia-gtc-stock-jensen-huang-keynote-blackwell/card/nvidia-is-more-than-700-billion-off-its-peak-market-cap-KOx1pr53zmwPrr8Jeo2X",
            "news_source": "Market Watch",
            "article_title": "Nvidia is more than $700 billion off its peak market cap"
        },
        {
            "link": "https://finance.yahoo.com/news/nvidias-next-big-moment-coming-145200617.html?fr=sycsrp_catchall",
            "news_source": "Quartz via Yahoo Finance",
            "article_title": "Nvidia's next big moment has arrived. Here's what analysts are saying"
        },
        {
            "link": "https://finance.yahoo.com/news/ultimate-guide-nvidia-strategies-buy-081000772.html?fr=sycsrp_catchall",
            "news_source": "Motley Fool via Yahoo Finance",
            "article_title": "The Ultimate Guide to Nvidia: Strategies to Buy, Hold, and Profit"
        },
        {
            "link": "https://finance.yahoo.com/news/nvidia-ceo-jensen-huang-could-110504683.html?fr=sycsrp_catchall",
            "news_source": "TheStreet via Yahoo Finance",
            "article_title": "Nvidia CEO Jensen Huang could reboot AI trade with GTC address"
        },
        {
            "link": "https://finance.yahoo.com/news/nvidia-google-tesla-byd-tencent-124200491.html?fr=sycsrp_catchall",
            "news_source": "Quartz via Yahoo Finance",
            "article_title": "Nvidia, Google, Tesla, BYD, Tencent Music, Xiaomi, Xpeng: Stocks to watch today"
        },
        {
            "link": "https://www.barrons.com/articles/stock-market-movers-dde42757?siteid=yhoof2&yptr=yahoo",
            "news_source": "Barrons.com",
            "article_title": "These Stocks Are Moving the Most Today: Nvidia, Intel, Alphabet, Tesla, XPeng, Sarepta, and More"
        },
        {
            "link": "https://finance.yahoo.com/news/nvidia-nvda-best-reddit-stock-111518863.html?fr=sycsrp_catchall",
            "news_source": "Insider Monkey via Yahoo Finance",
            "article_title": "Is NVIDIA (NVDA) the Best Reddit Stock to Invest In?"
        }
    ],
    "people_also_ask": [
        {
            "link": "https://www.nvidia.com/en-us/drivers/",
            "answer": "The NVIDIA App is the essential companion for PC gamers and creators. Keep your PC up to date with the latest NVIDIA drivers and technology. No Drivers Found. Please select a different combination. Note: Many Linux distributions provide their own packages of the NVIDIA Linux Graphics Driver in the distribution's native package management format.",
            "question": "What is the Nvidia app?"
        },
        {
            "link": "https://en.wikipedia.org/wiki/Nvidia",
            "answer": "Nvidia's shares traded at over $531 per share, and its market capitalization was valued at over US$328.7 billion in January 2021. As of late Q3 2024, Nvidia's market cap is around US$2.98 trillion. For the Q2 of 2020, Nvidia reported sales of $3.87 billion, which was a 50% rise from the same period in 2019.",
            "question": "How much is Nvidia worth?"
        },
        {
            "link": "https://nvidianews.nvidia.com/news/nvidia-delivers-quantum-leap-in-performance-introduces-new-era-of-neural-rendering-with-geforce-rtx-40-series",
            "answer": "The company\u2019s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined computer graphics and ignited the era of modern AI. NVIDIA is now a full-stack computing company with data-center-scale offerings that are reshaping industry. More information at",
            "question": "Why is Nvidia reshaping the gaming industry?"
        },
        {
            "link": "https://en.wikipedia.org/wiki/Nvidia",
            "answer": "Nvidia Corporation[a] (/ \u025bn\u02c8v\u026adi\u0259 / en-VID-ee-\u0259) is an American multinational corporation and technology company headquartered in Santa Clara, California, and incorporated in Delaware.",
            "question": "Where is Nvidia based?"
        },
        {
            "link": "https://en.wikipedia.org/wiki/Nvidia",
            "answer": "Nvidia outsources the manufacturing of the hardware it designs. Nvidia's professional line of GPUs are used for edge-to-cloud computing and in supercomputers and workstations for applications in fields such as architecture, engineering and construction, media and entertainment, automotive, scientific research, and manufacturing design.",
            "question": "What does Nvidia do for a living?"
        },
        {
            "link": "https://en.wikipedia.org/wiki/Nvidia",
            "answer": "Nvidia's product families include graphics processing units, wireless communication devices, and automotive hardware and software, such as: Drive, a range of hardware and software products for designers and manufacturers of autonomous vehicles.",
            "question": "What are Nvidia products?"
        }
    ],
    "related_searches": [
        "nvidia broadcast",
        "nvidia app",
        "nvidia graphics cards",
        "nvidia download",
        "nvidia geforce experience",
        "nvidia driver download",
        "nvidia stock prices today",
        "nvidia control panel"
    ],
    "organic_result_listings": [
        {
            "link": "http://www.nvidia.com/page/home.html",
            "title": "Nvidia - Official Site",
            "description": "NVIDIA is a company that develops and provides AI solutions for various industries and applications. Learn about its latest products, events, reports, and innovations in AI, simulation, HPC, and quantum computing."
        },
        {
            "link": "https://news.search.yahoo.com/search;_ylt=AwrEo62Bctln.QEAVSNXNyoA;_ylu=Y29sbwNiZjEEcG9zAzEEdnRpZAMEc2VjA3Nj?p=nvidia&fr2=p%3As%2Cv%3Aw%2Cm%3Anewsdd_sna_t%2Cct%3Anuwa",
            "title": "Top Stories",
            "description": null
        },
        {
            "link": "https://www.nvidia.com/en-us/drivers/",
            "title": "Download The Official NVIDIA Drivers | NVIDIA",
            "description": "Download the latest official NVIDIA drivers to enhance your PC gaming experience and run apps faster."
        },
        {
            "link": "https://www.nvidia.com/en-us/",
            "title": "World Leader in Artificial Intelligence Computing | NVIDIA",
            "description": "NVIDIA is a company that develops and provides AI solutions for various industries and applications. Learn about its latest products, events, reports, and innovations in AI, simulation, HPC, and quantum computing."
        },
        {
            "link": "https://video.search.yahoo.com/search/video;_ylt=AwrEo62Bctln.QEAciNXNyoA;_ylu=Y29sbwNiZjEEcG9zAzIEdnRpZAMEc2VjA3Nj?p=nvidia",
            "title": "Videos",
            "description": null
        },
        {
            "link": "https://en.wikipedia.org/wiki/Nvidia",
            "title": "Nvidia - Wikipedia",
            "description": "Nvidia Corporation [a] (/ \u025b n \u02c8 v \u026a d i \u0259 / en-VID-ee-\u0259) is an American multinational corporation and technology company headquartered in Santa Clara, California, and incorporated in Delaware. [5]"
        },
        {
            "link": "https://marketplace.nvidia.com/en-us/",
            "title": "NVIDIA Marketplace US: Graphics Cards, AI, Gaming ...",
            "description": "Shop for NVIDIA products and solutions for gaming, data center, robotics, edge, software, and more. Find qualified systems, networking, and online training for GeForce graphics cards and RTX workstations."
        },
        {
            "link": "https://marketplace.nvidia.com/en-us/consumer/gfn/",
            "title": "NVIDIA GeForce Graphics Cards, Gaming Laptops, Desktop and ...",
            "description": "Shop for NVIDIA GeForce graphics cards, gaming laptops, desktop and monitor from the official store. Explore the next generation in cloud gaming with GFN Cloud Gaming and RTX On."
        },
        {
            "link": "https://techcrunch.com/storyline/nvidia-gtc-2025-live-updates-blackwell-ultra-next-gen-rubin-chip-architecture-and-much-more/",
            "title": "Nvidia GTC 2025 live updates: Blackwell Ultra, next-gen Rubin ...",
            "description": "1 day ago \u00b7 Nvidia typically reserves a big chunk of the conference for GPU-related debuts. A new, upgraded iteration of the company\u2019s Blackwell chip lineup seems likely. Rubin, Nvidia\u2019s next-gen GPU ..."
        }
    ]
}

Schema

OxyCopilot builds a parsing schema that outlines the data structure, including object types, field names, and descriptions. The AI then uses this schema to generate instructions, which you can customize in the Web Scraper APIs Playground to suit your parsing needs.

Object type	Name	Description
Array of objects	organic_result_listings	List of organic result listings.
String	title	Extract text only from the <a> element.
String	description
String	link
Array of objects	top_stories	List of top stories.
String	article_title
String	link
String	news_source
Array of objects	people_also_ask	List of People Also Ask questions.
String	question
String	answer
String	link
Array of strings	related_searches	List of related searches extracted from the 'title' attributes.

If you've never used Web Scraper API before, sign up for a 1-week free trial or find a suitable plan on the dashboard to get started.

Claiming a free trial

Creating an API user

Next, you need to create your API user. Enter your preferred username, password, and click Create API User.

Save your credentials as you'll need them for future Web Scraper API and OxyCopilot use.

We also have a video tutorial, explaining how to get a free trial, create an API user, and more:

Scraper APIs Playground

Once the setup is complete, click the Scraper APIs Playground button on the left. You'll be redirected to the playground page, where you can start using OxyCopilot.

Now that you're on the Scraper APIs Playground, click the OxyCopilot button in the top-left corner to begin creating your scraper and parser for collecting search results data from Yahoo!.

Using OxyCopilot

After clicking this button, you'll see the table, where you need to enter URLs, Scraper Parameters, and Parsing instructions.

1. Enter URL(s) to scrape

You can use any Yahoo! URL to scrape and extract data from. But for the purpose of this tutorial, we'll parse search results for the term “nvidia”:

Copy

https://search.yahoo.com/search?p=nvidia

2. Scraper parameters

Select your API parameters, including location settings. Be sure to enable JavaScript rendering for pages with dynamic content to ensure complete data extraction. This example shows OxyCopilot configured to use United States geolocation:

Copy

Scrape this URL from the United States location.

3. Enter parsing instructions prompt

Next, create and submit a prompt that will generate the necessary parsing instructions. The example below instructs OxyCopilot what kind of information to extract from Yahoo! search results page.

Aim to maintain a well-structured and clean prompt to get the best results. For complex queries, you may need to resubmit your prompt to achieve the desired outcome. If the page contains intricate HTML structures, you can enhance accuracy by explicitly specifying which HTML elements or attributes to target.

Copy

Parse the following sections from a search results page:
- Organic result listings:
  -- Title: Extract text only from the <a> element.
  -- Description.
  -- Link.
- Top Stories. From each card in the list, extract the following:
  -- Article title.
  -- Link.
  -- News source.
- People Also Ask: Parse the question, answer, and link of each FAQ.
- Related Searches list: Extract the values from the "title" attributes.

Generated instructions

When executed, the prompt generates JSON-formatted parsing instructions that integrate with the API's Custom Parser functionality.

Store these instructions separately for your API requests, or click the "Save Instructions" button and then copy the full request code in your preferred language. For our recommended approach, check the Results tab.

Copy

{
    "organic_result_listings": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": [
                    "//ol[@class=\"reg  searchCenterMiddle\"]/li",
                    "//div[@class=\"web-res\"]/ol/li",
                    "//div[@id=\"web\"]/ol/li",
                    "//div[@class=\"main-res\"]/div/div/ol/li",
                    "//div[@id=\"main\"]/div/div/ol/li",
                    "//div[@id=\"left\"]/div/div/div/div/ol/li",
                    "//div[@id=\"cols\"]/div[1]/div/div/div/div/ol/li",
                    "//div[@id=\"results\"]/div/div[1]/div/div/div/div/ol/li",
                    "//div[@role=\"document\"]/div/div/div[1]/div/div/div/div/ol/li",
                    "//div[@id=\"bd\"]/div/div/div[1]/div/div/div/div/ol/li"
                ]
            }
        ],
        "_items": {
            "title": {
                "_fns": [
                    {
                        "_fn": "xpath_one",
                        "_args": [
                            ".//a[@data-matarget=\"algo\"]/text()",
                            ".//a[@class=\"d-ib fz-20 ls-024 lh-19 td-hu tc va-bot mxw-100p mt-8\"]/text()",
                            ".//h3[@class=\"title mt-15 mb-4\"]/a/text()",
                            ".//div[@class=\"compTitle options-toggle\"]/h3/a/text()",
                            ".//div/div[1]/h3/a/text()",
                            ".//div[@class=\"dd algo algo-sr relsrch Sr\"]/div[1]/h3/a/text()"
                        ]
                    },
                    {
                        "_fn": "regex_search",
                        "_args": [
                            "^\\s*(.[\\s\\S]*?)\\s*$",
                            1
                        ]
                    }
                ]
            },
            "description": {
                "_fns": [
                    {
                        "_fn": "xpath",
                        "_args": [
                            ".//p[@class=\"fc-dustygray fz-14 lh-22 ls-02 mah-44 ov-h d-box fbox-ov fbox-lc2\"]",
                            ".//div[@class=\"compText aAbs\"]/p",
                            ".//div/div[2]/p",
                            ".//div[@class=\"dd algo algo-sr relsrch Sr\"]/div[2]/p"
                        ]
                    },
                    {
                        "_fn": "xpath",
                        "_args": [
                            "normalize-space(.)"
                        ]
                    },
                    {
                        "_fn": "join",
                        "_args": " "
                    }
                ]
            },
            "link": {
                "_fns": [
                    {
                        "_fn": "xpath_one",
                        "_args": [
                            ".//a[@data-matarget=\"algo\"]/@href",
                            ".//a[@class=\"d-ib fz-20 ls-024 lh-19 td-hu tc va-bot mxw-100p mt-8\"]/@href",
                            ".//h3[@class=\"title mt-15 mb-4\"]/a/@href",
                            ".//div[@class=\"compTitle options-toggle\"]/h3/a/@href",
                            ".//div/div[1]/h3/a/@href",
                            ".//div[@class=\"dd algo algo-sr relsrch Sr\"]/div[1]/h3/a/@href"
                        ]
                    },
                    {
                        "_fn": "regex_search",
                        "_args": [
                            "^\\s*(.[\\s\\S]*?)\\s*$",
                            1
                        ]
                    }
                ]
            }
        }
    },
    "top_stories": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": [
                    "//div[contains(@class, 'dd tn-carousel')]//li[contains(@class, 's-card')]"
                ]
            }
        ],
        "_items": {
            "article_title": {
                "_fns": [
                    {
                        "_fn": "xpath_one",
                        "_args": [
                            ".//a/div[contains(@class, 's-card-bot')]/p[2]/span/text()"
                        ]
                    },
                    {
                        "_fn": "regex_search",
                        "_args": [
                            "^\\s*(.[\\s\\S]*?)\\s*$",
                            1
                        ]
                    }
                ]
            },
            "link": {
                "_fns": [
                    {
                        "_fn": "xpath_one",
                        "_args": [
                            ".//a/@href"
                        ]
                    },
                    {
                        "_fn": "regex_search",
                        "_args": [
                            "^\\s*(.[\\s\\S]*?)\\s*$",
                            1
                        ]
                    }
                ]
            },
            "news_source": {
                "_fns": [
                    {
                        "_fn": "xpath_one",
                        "_args": [
                            ".//a/div[contains(@class, 's-card-bot')]/p[1]/span/text()"
                        ]
                    },
                    {
                        "_fn": "regex_search",
                        "_args": [
                            "^\\s*(.[\\s\\S]*?)\\s*$",
                            1
                        ]
                    }
                ]
            }
        }
    },
    "people_also_ask": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": [
                    "//div[@class=\"compContainerExp compContainerExpRight\"]/div",
                    "//div[@class=\"dd bb-1 bingrelqa\"]/div[2]/div",
                    "//ol[@class=\"reg  searchCenterMiddle\"]/li[10]/div/div[2]/div",
                    "//div[@class=\"web-res\"]/ol/li[10]/div/div[2]/div",
                    "//div[@id=\"web\"]/ol/li[10]/div/div[2]/div",
                    "//div[@class=\"main-res\"]/div/div/ol/li[10]/div/div[2]/div",
                    "//div[@id=\"main\"]/div/div/ol/li[10]/div/div[2]/div",
                    "//div[@id=\"left\"]/div/div/div/div/ol/li[10]/div/div[2]/div",
                    "//div[@id=\"cols\"]/div[1]/div/div/div/div/ol/li[10]/div/div[2]/div",
                    "//div[@id=\"results\"]/div/div[1]/div/div/div/div/ol/li[10]/div/div[2]/div"
                ]
            }
        ],
        "_items": {
            "question": {
                "_fns": [
                    {
                        "_fn": "xpath_one",
                        "_args": [
                            ".//span[@class=\"cur-p fc-refblack fz-16 fw-500 lh-28 ls-027 va-top pr-36\"]/text()",
                            ".//div[@class=\"compText\"]/p/span/text()",
                            ".//div[@class=\"ttl-l\"]/div/p/span/text()",
                            ".//div[@role=\"button\"]/div/p/span/text()",
                            ".//div[@aria-expanded=\"false\"]/div/p/span/text()",
                            ".//div[@class=\"ttl \"]/div[2]/div/p/span/text()",
                            ".//div[@class=\"cnt mt-0\"]/div[1]/div[2]/div/p/span/text()"
                        ]
                    },
                    {
                        "_fn": "regex_search",
                        "_args": [
                            "^\\s*(.[\\s\\S]*?)\\s*$",
                            1
                        ]
                    }
                ]
            },
            "answer": {
                "_fns": [
                    {
                        "_fn": "xpath",
                        "_args": [
                            ".//div[@class=\"lh-m ls-024 d-b \"]",
                            ".//li[@role=\"group\"]/div",
                            ".//li[@class=\"va-top mb-8 fz-m\"]/div",
                            ".//li[@aria-label=\"Item 1 of 1\"]/div",
                            ".//ul[@role=\"region\"]/li/div",
                            ".//ul[@aria-label=\"carousel\"]/li/div",
                            ".//div[@class=\"compTextList\"]/ul/li/div"
                        ]
                    },
                    {
                        "_fn": "xpath",
                        "_args": [
                            "normalize-space(.)"
                        ]
                    },
                    {
                        "_fn": "join",
                        "_args": " "
                    }
                ]
            },
            "link": {
                "_fns": [
                    {
                        "_fn": "xpath_one",
                        "_args": [
                            ".//a[@target=\"_blank\"]/@href",
                            ".//a[@referrerpolicy=\"origin\"]/@href",
                            ".//h3[@class=\"title\"]/a/@href",
                            ".//div[@class=\"compTitle pt-10 pb-15\"]/h3/a/@href",
                            ".//div[@class=\"exp hidden \"]/div[2]/h3/a/@href",
                            ".//div[@class=\"cnt mt-0\"]/div[2]/div[2]/h3/a/@href",
                            ".//div/div[2]/div[2]/h3/a/@href"
                        ]
                    },
                    {
                        "_fn": "regex_search",
                        "_args": [
                            "^\\s*(.[\\s\\S]*?)\\s*$",
                            1
                        ]
                    }
                ]
            }
        }
    },
    "related_searches": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": [
                    "//div[@id=\"bd\"]/div/div/div[1]/div/div/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title",
                    "//div[@id=\"web\"]/ol/li[3]/div/table/tbody//tr/td/a/@title",
                    "//div[@id=\"left\"]/div/div/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title",
                    "//div[@id=\"cols\"]/div[1]/div/div/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title",
                    "//div[@id=\"main\"]/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title",
                    "//div[@id=\"results\"]/div/div[1]/div/div/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title",
                    "//div[@class=\"dd AlsoTry_M\"]/table/tbody//tr/td/a/@title",
                    "//div[@class=\"main-res\"]/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title",
                    "//div[@class=\"web-res\"]/ol/li[3]/div/table/tbody//tr/td/a/@title",
                    "//ol[@class=\"reg  searchCenterMiddle\"]/li[3]/div/table/tbody//tr/td/a/@title",
                    "/html/body/div[1]/div[3]/div/div/div[1]/div/div/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title"
                ]
            },
            {
                "_fn": "regex_search",
                "_args": [
                    "^\\s*(.[\\s\\S]*?)\\s*$",
                    1
                ]
            }
        ]
    }
}

Input

Here's the generated API request code you can use in your preferred IDE to gather Yahoo! search results data. Remember to enter the credentials you've got after creating your API user because you won't be able to use our Web Scraper API otherwise.

Copy

import requests
from pprint import pprint

# Structure payload.
payload = {
   'source': 'universal',
   'url': 'https://search.yahoo.com/search?p=nvidia',
   'geo_location': 'United States',
   'render': 'html',
   'parse': True,
   'parsing_instructions': {"organic_result_listings":{"_fns":[{"_fn":"xpath","_args":["//ol[@class=\"reg  searchCenterMiddle\"]/li","//div[@class=\"web-res\"]/ol/li","//div[@id=\"web\"]/ol/li","//div[@class=\"main-res\"]/div/div/ol/li","//div[@id=\"main\"]/div/div/ol/li","//div[@id=\"left\"]/div/div/div/div/ol/li","//div[@id=\"cols\"]/div[1]/div/div/div/div/ol/li","//div[@id=\"results\"]/div/div[1]/div/div/div/div/ol/li","//div[@role=\"document\"]/div/div/div[1]/div/div/div/div/ol/li","//div[@id=\"bd\"]/div/div/div[1]/div/div/div/div/ol/li"]}],"_items":{"title":{"_fns":[{"_fn":"xpath_one","_args":[".//a[@data-matarget=\"algo\"]/text()",".//a[@class=\"d-ib fz-20 ls-024 lh-19 td-hu tc va-bot mxw-100p mt-8\"]/text()",".//h3[@class=\"title mt-15 mb-4\"]/a/text()",".//div[@class=\"compTitle options-toggle\"]/h3/a/text()",".//div/div[1]/h3/a/text()",".//div[@class=\"dd algo algo-sr relsrch Sr\"]/div[1]/h3/a/text()"]},{"_fn":"regex_search","_args":["^\\s*(.[\\s\\S]*?)\\s*$",1]}]},"description":{"_fns":[{"_fn":"xpath","_args":[".//p[@class=\"fc-dustygray fz-14 lh-22 ls-02 mah-44 ov-h d-box fbox-ov fbox-lc2\"]",".//div[@class=\"compText aAbs\"]/p",".//div/div[2]/p",".//div[@class=\"dd algo algo-sr relsrch Sr\"]/div[2]/p"]},{"_fn":"xpath","_args":["normalize-space(.)"]},{"_fn":"join","_args":" "}]},"link":{"_fns":[{"_fn":"xpath_one","_args":[".//a[@data-matarget=\"algo\"]/@href",".//a[@class=\"d-ib fz-20 ls-024 lh-19 td-hu tc va-bot mxw-100p mt-8\"]/@href",".//h3[@class=\"title mt-15 mb-4\"]/a/@href",".//div[@class=\"compTitle options-toggle\"]/h3/a/@href",".//div/div[1]/h3/a/@href",".//div[@class=\"dd algo algo-sr relsrch Sr\"]/div[1]/h3/a/@href"]},{"_fn":"regex_search","_args":["^\\s*(.[\\s\\S]*?)\\s*$",1]}]}}},"top_stories":{"_fns":[{"_fn":"xpath","_args":["//div[contains(@class, 'dd tn-carousel')]//li[contains(@class, 's-card')]"]}],"_items":{"article_title":{"_fns":[{"_fn":"xpath_one","_args":[".//a/div[contains(@class, 's-card-bot')]/p[2]/span/text()"]},{"_fn":"regex_search","_args":["^\\s*(.[\\s\\S]*?)\\s*$",1]}]},"link":{"_fns":[{"_fn":"xpath_one","_args":[".//a/@href"]},{"_fn":"regex_search","_args":["^\\s*(.[\\s\\S]*?)\\s*$",1]}]},"news_source":{"_fns":[{"_fn":"xpath_one","_args":[".//a/div[contains(@class, 's-card-bot')]/p[1]/span/text()"]},{"_fn":"regex_search","_args":["^\\s*(.[\\s\\S]*?)\\s*$",1]}]}}},"people_also_ask":{"_fns":[{"_fn":"xpath","_args":["//div[@class=\"compContainerExp compContainerExpRight\"]/div","//div[@class=\"dd bb-1 bingrelqa\"]/div[2]/div","//ol[@class=\"reg  searchCenterMiddle\"]/li[10]/div/div[2]/div","//div[@class=\"web-res\"]/ol/li[10]/div/div[2]/div","//div[@id=\"web\"]/ol/li[10]/div/div[2]/div","//div[@class=\"main-res\"]/div/div/ol/li[10]/div/div[2]/div","//div[@id=\"main\"]/div/div/ol/li[10]/div/div[2]/div","//div[@id=\"left\"]/div/div/div/div/ol/li[10]/div/div[2]/div","//div[@id=\"cols\"]/div[1]/div/div/div/div/ol/li[10]/div/div[2]/div","//div[@id=\"results\"]/div/div[1]/div/div/div/div/ol/li[10]/div/div[2]/div"]}],"_items":{"question":{"_fns":[{"_fn":"xpath_one","_args":[".//span[@class=\"cur-p fc-refblack fz-16 fw-500 lh-28 ls-027 va-top pr-36\"]/text()",".//div[@class=\"compText\"]/p/span/text()",".//div[@class=\"ttl-l\"]/div/p/span/text()",".//div[@role=\"button\"]/div/p/span/text()",".//div[@aria-expanded=\"false\"]/div/p/span/text()",".//div[@class=\"ttl \"]/div[2]/div/p/span/text()",".//div[@class=\"cnt mt-0\"]/div[1]/div[2]/div/p/span/text()"]},{"_fn":"regex_search","_args":["^\\s*(.[\\s\\S]*?)\\s*$",1]}]},"answer":{"_fns":[{"_fn":"xpath","_args":[".//div[@class=\"lh-m ls-024 d-b \"]",".//li[@role=\"group\"]/div",".//li[@class=\"va-top mb-8 fz-m\"]/div",".//li[@aria-label=\"Item 1 of 1\"]/div",".//ul[@role=\"region\"]/li/div",".//ul[@aria-label=\"carousel\"]/li/div",".//div[@class=\"compTextList\"]/ul/li/div"]},{"_fn":"xpath","_args":["normalize-space(.)"]},{"_fn":"join","_args":" "}]},"link":{"_fns":[{"_fn":"xpath_one","_args":[".//a[@target=\"_blank\"]/@href",".//a[@referrerpolicy=\"origin\"]/@href",".//h3[@class=\"title\"]/a/@href",".//div[@class=\"compTitle pt-10 pb-15\"]/h3/a/@href",".//div[@class=\"exp hidden \"]/div[2]/h3/a/@href",".//div[@class=\"cnt mt-0\"]/div[2]/div[2]/h3/a/@href",".//div/div[2]/div[2]/h3/a/@href"]},{"_fn":"regex_search","_args":["^\\s*(.[\\s\\S]*?)\\s*$",1]}]}}},"related_searches":{"_fns":[{"_fn":"xpath","_args":["//div[@id=\"bd\"]/div/div/div[1]/div/div/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title","//div[@id=\"web\"]/ol/li[3]/div/table/tbody//tr/td/a/@title","//div[@id=\"left\"]/div/div/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title","//div[@id=\"cols\"]/div[1]/div/div/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title","//div[@id=\"main\"]/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title","//div[@id=\"results\"]/div/div[1]/div/div/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title","//div[@class=\"dd AlsoTry_M\"]/table/tbody//tr/td/a/@title","//div[@class=\"main-res\"]/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title","//div[@class=\"web-res\"]/ol/li[3]/div/table/tbody//tr/td/a/@title","//ol[@class=\"reg  searchCenterMiddle\"]/li[3]/div/table/tbody//tr/td/a/@title","/html/body/div[1]/div[3]/div/div/div[1]/div/div/div/div/ol/li[3]/div/table/tbody//tr/td/a/@title"]},{"_fn":"regex_search","_args":["^\\s*(.[\\s\\S]*?)\\s*$",1]}]}}
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('YOUR_USERNAME', 'YOUR_PASSWORD'), # Your credentials go here
    json=payload,
)

# Instead of response with job status and results url, this will return the
# JSON response with results.
pprint(response.json())

Discover more useful information on Web Scraper API, such as parameter values, integration methods or features in our comprehensive documentation.

Output

The API response will return structured Yahoo! data based on the parsing instructions, as shown below:

Copy

{
    "top_stories": [
        {
            "link": "https://www.proactiveinvestors.com/companies/news/1068066/nvidia-gtc-conference-a-wake-up-call-for-tech-bulls-says-investment-bank-1068066.html",
            "news_source": "Proactive Investors",
            "article_title": "Nvidia GTC Conference a 'wake-up call' for tech bulls, says investment bank"
        },
        {
            "link": "https://finance.yahoo.com/news/why-nvidia-stock-holds-the-key-to-unlocking-a-rejuvenated-market-123646743.html?fr=sycsrp_catchall",
            "news_source": "Yahoo Finance",
            "article_title": "Why Nvidia stock holds the key to unlocking a rejuvenated market"
        },
        {
            "link": "https://finance.yahoo.com/news/where-nvidia-1-113000389.html?fr=sycsrp_catchall",
            "news_source": "Motley Fool via Yahoo Finance",
            "article_title": "Where Will Nvidia Be in 1 Year?"
        },
        {
            "link": "https://www.marketwatch.com/livecoverage/nvidia-gtc-stock-jensen-huang-keynote-blackwell/card/nvidia-is-more-than-700-billion-off-its-peak-market-cap-KOx1pr53zmwPrr8Jeo2X",
            "news_source": "Market Watch",
            "article_title": "Nvidia is more than $700 billion off its peak market cap"
        },
        {
            "link": "https://finance.yahoo.com/news/nvidias-next-big-moment-coming-145200617.html?fr=sycsrp_catchall",
            "news_source": "Quartz via Yahoo Finance",
            "article_title": "Nvidia's next big moment has arrived. Here's what analysts are saying"
        },
        {
            "link": "https://finance.yahoo.com/news/ultimate-guide-nvidia-strategies-buy-081000772.html?fr=sycsrp_catchall",
            "news_source": "Motley Fool via Yahoo Finance",
            "article_title": "The Ultimate Guide to Nvidia: Strategies to Buy, Hold, and Profit"
        },
        {
            "link": "https://finance.yahoo.com/news/nvidia-ceo-jensen-huang-could-110504683.html?fr=sycsrp_catchall",
            "news_source": "TheStreet via Yahoo Finance",
            "article_title": "Nvidia CEO Jensen Huang could reboot AI trade with GTC address"
        },
        {
            "link": "https://finance.yahoo.com/news/nvidia-google-tesla-byd-tencent-124200491.html?fr=sycsrp_catchall",
            "news_source": "Quartz via Yahoo Finance",
            "article_title": "Nvidia, Google, Tesla, BYD, Tencent Music, Xiaomi, Xpeng: Stocks to watch today"
        },
        {
            "link": "https://www.barrons.com/articles/stock-market-movers-dde42757?siteid=yhoof2&yptr=yahoo",
            "news_source": "Barrons.com",
            "article_title": "These Stocks Are Moving the Most Today: Nvidia, Intel, Alphabet, Tesla, XPeng, Sarepta, and More"
        },
        {
            "link": "https://finance.yahoo.com/news/nvidia-nvda-best-reddit-stock-111518863.html?fr=sycsrp_catchall",
            "news_source": "Insider Monkey via Yahoo Finance",
            "article_title": "Is NVIDIA (NVDA) the Best Reddit Stock to Invest In?"
        }
    ],
    "people_also_ask": [
        {
            "link": "https://www.nvidia.com/en-us/drivers/",
            "answer": "The NVIDIA App is the essential companion for PC gamers and creators. Keep your PC up to date with the latest NVIDIA drivers and technology. No Drivers Found. Please select a different combination. Note: Many Linux distributions provide their own packages of the NVIDIA Linux Graphics Driver in the distribution's native package management format.",
            "question": "What is the Nvidia app?"
        },
        {
            "link": "https://en.wikipedia.org/wiki/Nvidia",
            "answer": "Nvidia's shares traded at over $531 per share, and its market capitalization was valued at over US$328.7 billion in January 2021. As of late Q3 2024, Nvidia's market cap is around US$2.98 trillion. For the Q2 of 2020, Nvidia reported sales of $3.87 billion, which was a 50% rise from the same period in 2019.",
            "question": "How much is Nvidia worth?"
        },
        {
            "link": "https://nvidianews.nvidia.com/news/nvidia-delivers-quantum-leap-in-performance-introduces-new-era-of-neural-rendering-with-geforce-rtx-40-series",
            "answer": "The company\u2019s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined computer graphics and ignited the era of modern AI. NVIDIA is now a full-stack computing company with data-center-scale offerings that are reshaping industry. More information at",
            "question": "Why is Nvidia reshaping the gaming industry?"
        },
        {
            "link": "https://en.wikipedia.org/wiki/Nvidia",
            "answer": "Nvidia Corporation[a] (/ \u025bn\u02c8v\u026adi\u0259 / en-VID-ee-\u0259) is an American multinational corporation and technology company headquartered in Santa Clara, California, and incorporated in Delaware.",
            "question": "Where is Nvidia based?"
        },
        {
            "link": "https://en.wikipedia.org/wiki/Nvidia",
            "answer": "Nvidia outsources the manufacturing of the hardware it designs. Nvidia's professional line of GPUs are used for edge-to-cloud computing and in supercomputers and workstations for applications in fields such as architecture, engineering and construction, media and entertainment, automotive, scientific research, and manufacturing design.",
            "question": "What does Nvidia do for a living?"
        },
        {
            "link": "https://en.wikipedia.org/wiki/Nvidia",
            "answer": "Nvidia's product families include graphics processing units, wireless communication devices, and automotive hardware and software, such as: Drive, a range of hardware and software products for designers and manufacturers of autonomous vehicles.",
            "question": "What are Nvidia products?"
        }
    ],
    "related_searches": [
        "nvidia broadcast",
        "nvidia app",
        "nvidia graphics cards",
        "nvidia download",
        "nvidia geforce experience",
        "nvidia driver download",
        "nvidia stock prices today",
        "nvidia control panel"
    ],
    "organic_result_listings": [
        {
            "link": "http://www.nvidia.com/page/home.html",
            "title": "Nvidia - Official Site",
            "description": "NVIDIA is a company that develops and provides AI solutions for various industries and applications. Learn about its latest products, events, reports, and innovations in AI, simulation, HPC, and quantum computing."
        },
        {
            "link": "https://news.search.yahoo.com/search;_ylt=AwrEo62Bctln.QEAVSNXNyoA;_ylu=Y29sbwNiZjEEcG9zAzEEdnRpZAMEc2VjA3Nj?p=nvidia&fr2=p%3As%2Cv%3Aw%2Cm%3Anewsdd_sna_t%2Cct%3Anuwa",
            "title": "Top Stories",
            "description": null
        },
        {
            "link": "https://www.nvidia.com/en-us/drivers/",
            "title": "Download The Official NVIDIA Drivers | NVIDIA",
            "description": "Download the latest official NVIDIA drivers to enhance your PC gaming experience and run apps faster."
        },
        {
            "link": "https://www.nvidia.com/en-us/",
            "title": "World Leader in Artificial Intelligence Computing | NVIDIA",
            "description": "NVIDIA is a company that develops and provides AI solutions for various industries and applications. Learn about its latest products, events, reports, and innovations in AI, simulation, HPC, and quantum computing."
        },
        {
            "link": "https://video.search.yahoo.com/search/video;_ylt=AwrEo62Bctln.QEAciNXNyoA;_ylu=Y29sbwNiZjEEcG9zAzIEdnRpZAMEc2VjA3Nj?p=nvidia",
            "title": "Videos",
            "description": null
        },
        {
            "link": "https://en.wikipedia.org/wiki/Nvidia",
            "title": "Nvidia - Wikipedia",
            "description": "Nvidia Corporation [a] (/ \u025b n \u02c8 v \u026a d i \u0259 / en-VID-ee-\u0259) is an American multinational corporation and technology company headquartered in Santa Clara, California, and incorporated in Delaware. [5]"
        },
        {
            "link": "https://marketplace.nvidia.com/en-us/",
            "title": "NVIDIA Marketplace US: Graphics Cards, AI, Gaming ...",
            "description": "Shop for NVIDIA products and solutions for gaming, data center, robotics, edge, software, and more. Find qualified systems, networking, and online training for GeForce graphics cards and RTX workstations."
        },
        {
            "link": "https://marketplace.nvidia.com/en-us/consumer/gfn/",
            "title": "NVIDIA GeForce Graphics Cards, Gaming Laptops, Desktop and ...",
            "description": "Shop for NVIDIA GeForce graphics cards, gaming laptops, desktop and monitor from the official store. Explore the next generation in cloud gaming with GFN Cloud Gaming and RTX On."
        },
        {
            "link": "https://techcrunch.com/storyline/nvidia-gtc-2025-live-updates-blackwell-ultra-next-gen-rubin-chip-architecture-and-much-more/",
            "title": "Nvidia GTC 2025 live updates: Blackwell Ultra, next-gen Rubin ...",
            "description": "1 day ago \u00b7 Nvidia typically reserves a big chunk of the conference for GPU-related debuts. A new, upgraded iteration of the company\u2019s Blackwell chip lineup seems likely. Rubin, Nvidia\u2019s next-gen GPU ..."
        }
    ]
}

Schema

Object type	Name	Description
Array of objects	organic_result_listings	List of organic result listings.
String	title	Extract text only from the <a> element.
String	description
String	link
Array of objects	top_stories	List of top stories.
String	article_title
String	link
String	news_source
Array of objects	people_also_ask	List of People Also Ask questions.
String	question
String	answer
String	link
Array of strings	related_searches	List of related searches extracted from the 'title' attributes.