Web Scraping With n8n and Oxylabs

1. Add a trigger

The first step is to add a trigger that defines how and when the workflow should be executed. For this tutorial, let’s keep it simple by adding a Trigger manually step.

2. Add the HTTP Request node

Click the + sign, search for HTTP Request, and add it to the workflow.

3. Select a method and an API endpoint

Set up the HTTP Request node as follows:

Method: POST

URL: https://realtime.oxylabs.io/v1/queries 

You can also use the Push-Pull integration, which requires adding a few extra HTTP Request nodes to check the job status and retrieve results. If you choose this method, make sure you set the correct HTTP request method as outlined in the documentation.

4. Set up authentication

Next, set up API authentication as shown below:

Authentication: Generic Credential Type

Generic Auth Type: Basic Auth

Basic Auth: Create new credential

5. Create the API payload

Next, enable Send Body and select the following options:

Body Content Type: JSON

Specify Body: Using Fields Below or Using JSON

If you select the Using Fields Below option, you can manually enter all the API parameter names and values. The Using JSON option simplifies this process and lets you paste a JSON configuration, for example:

{
    "source": "google_search",
    "query": "best sportswear brands",
    "geo_location": "United States",
    "render": "html",
    "user_agent_type": "desktop",
    "parse": true
}

The "parse": true setting automatically parses the HTML code. For this reason, you don't need the HTML extract node for further processing. You may also use the Custom Parser feature that utilizes XPath and CSS selector expressions.

Integrating AI Studio

Oxylabs AI Studio is a low-code platform that enables you to build powerful scraping processes without the usual complexities, all while using simple English prompts to define your data needs.

Currently, the AI Studio node in n8n supports these AI applications:

• AI-Scraper

• AI-Crawler

• Browser Agent

• AI-Search

Don’t miss out and get your free API key with 1000 tokens by creating an account on the AI Studio website.

1. Install Oxylabs AI Studio node

As with the Web Scraper API setup, start the workflow with a manual trigger.

Then, click +, search for Oxylabs, and install the Oxylabs AI Studio node. After installation, add it to the workflow.

2. Enter your API key

Open the node and, in the Credential to connect with field, select Create new credential. In the new window, enter your AI Studio API key.

3. Select the AI Studio app

For demonstration purposes, let’s select the Scraper resource. Next, enter the URL you want to scrape and parse, for instance:

https://finance.yahoo.com/markets/stocks/gainers/

So far, your node should look like this:

4. Choose the output format

For the Output Format, you can choose between two options:

• Markdown – the default option, converts the entire page into a structured, LLM-friendly Markdown format.

• JSON – extracts specific data points as key-value pairs.

The JSON option requires a schema that defines the data to extract. You can quickly generate a schema in AI Studio by describing the data points in plain English. Simply enter the target URL, enable the Render JavaScript option, select JSON as the output format, and use a prompt like this:

Extract the stocks table. For each stock, get the symbol, name, price, change, change %, volume, avg volume, market cap, P/E ratio, and 52 Wk change %.

2. Add your OpenAI API key

Just like with other n8n nodes, setting up authentication is quick and straightforward. If you don’t have an OpenAI API key, you can create an OpenAI account and generate a key for free on the dashboard.

3. Set up AI for data processing

In your AI node, configure the following options:

Resource: Text

Model: GPT-4.1-NANO (pick any model that suits your needs)

Then, enter your prompt and include the scraped data in it by wrapping $json.results[0].content in {{ }}. Make sure to use the .toJsonString() function to convert the entire JSON object into a readable string, for example:

Analyze the key findings from Google results, provide a short and structured answer:
{{ $json.results[0].content.toJsonString() }}

Here’s the complete setup with the AI’s response: