Proxy locations

Europe

North America

South America

Asia

Africa

Oceania

See all locations

Network statusCareers

How to Set Up a Proxy Server on Screaming Frog

Building a solid SEO strategy requires fresh and accurate search engine results data. One way to get this data in bulk is by using web scraping tools and proxies. Proxies are inseparable from web scraping and help you bypass certain anti-bot measures; scraping public data is no exception. Read this Screaming Frog server integration guide and see how to set up Oxylabs Residential and Datacenter Proxies with this popular SEO Spider tool.

What is Screaming Frog?

Screaming Frog is a platform designed to scrape public web data for SEO-related purposes. It's available on macOS, Linux, and Windows devices. The tool offers public data scraping from up to 500 URLs; if you need more, you’ll need to upgrade to a bigger plan. 

Here are some common SEO-related initiatives professionals use this tool for: 

  • Redirect searches

  • Broken link searches

  • Content duplicate searches

  • Metadata analysis

  • Site architecture analysis

Screaming Frog SEO Spider proxy integration 

Before starting with the integration, download and install the program. If you have the paid version, navigate to the menu bar above and click Licence > Enter Licence. 

Screaming Frog licence

Activating a licence

A window will pop up, requiring you to enter your username and license key. When done, click OK and Restart. You’ll have received these details in your email upon registration. 

Screaming Frog licence

Activating a licence #2

1. Navigate to File > Settings > Proxy.

Screaming Frog proxy details

Navigating to proxy config

2. Insert your proxy details. Make sure to tick the Use Proxy Server box. 

Residential Proxies

Address: pr.oxylabs.io

Port: 7777

You can also connect to a proxy in a specific country. For example, if you enter us-pr.oxylabs.io under Address and 10000 under Port, you’ll connect to a US-based proxy server. Please refer to our documentation for a complete list of country-specific entry nodes or if you need a sticky session.

Screaming Frog proxy setup

Residential Proxies setup

Enterprise Dedicated Datacenter Proxies

Specify the following if you purchased Dedicated Datacenter Proxies via sales.

Address: 1.2.3.4 (a specific IP address)

Port: 60000

You’ll have to choose an IP address from the acquired list. Visit our documentation for more details.

Self-Service Dedicated Datacenter Proxies

Specify the following if you purchased Dedicated Datacenter Proxies via the dashboard.

Address: ddc.oxylabs.io

Port: 8001

Here, the port indicates the sequential number of an IP address from the acquired list. Check our documentation for more details.

Datacenter Proxies

Address: dc.oxylabs.io

Port: 8001

In the pay-per-IP subscription model, the port represents the sequential number assigned to an IP address from the given list, meaning port 8001 will use the first IP address on your proxy list. For more details, please check our documentation.

With the pay-per-traffic subscription, the 8001 port will pick a random IP address, but the IP will remain consistent for the duration of the session. You can also specify the proxy’s geo-location within the user authentication string, such as user-USERNAME-country-US:PASSWORD, which will connect you to a United States proxy. Please refer to our documentation for more information.

ISP Proxies

Address: isp.oxylabs.io

Port: 8001

3. Click OK and Restart to complete the proxy setup.

4. Upon starting a new crawling job, you'll be asked to enter your Oxylabs proxy user’s credentials. Use the same username and password you entered to create a new proxy user in the Oxylabs dashboard.

Screaming Frog proxy authentication

Authenticating proxy connection

That’s it! You can now start scraping public data without worrying about blocks.

Wrapping up  

Combined with Oxylabs proxies, Screaming Frog can deliver public data in real-time while bypassing anti-bot measures. 

As the tutorial shows, setting up proxies on this platform is simple, so you can start working on your SEO projects right away. If you have questions about integrating Oxylabs products, don’t hesitate to drop us a message at support@oxylabs.io or reach out via live chat. 

Please be aware that this is a third-party tool not owned or controlled by Oxylabs. Each third-party provider is responsible for its own software and services. Consequently, Oxylabs will have no liability or responsibility to you regarding those services. Please carefully review the third party's policies and practices and/or conduct due diligence before accessing or using third-party services.

Get the latest news from data gathering world

I'm interested