How to Set Up a Proxy Server on Screaming Frog
Building a solid SEO strategy requires fresh and accurate search engine results data. One way to get this data in bulk is by using web scraping tools and proxies. Proxies are inseparable from web scraping and help you bypass certain anti-bot measures; scraping public data is no exception. Read this Screaming Frog server integration guide and see how to set up Oxylabs Residential and Datacenter Proxies with this popular SEO Spider tool.
What is Screaming Frog?
Screaming Frog is a platform designed to scrape public web data for SEO-related purposes. It's available on macOS, Linux, and Windows devices. The tool offers public data scraping from up to 500 URLs; if you need more, you’ll need to upgrade to a bigger plan.
Here are some common SEO-related initiatives professionals use this tool for:
Redirect searches
Broken link searches
Content duplicate searches
Metadata analysis
Site architecture analysis
Screaming Frog SEO Spider proxy integration
Before starting with the integration, download and install the program. If you have the paid version, navigate to the menu bar above and click Licence > Enter Licence.
Activating a licence
A window will pop up, requiring you to enter your username and license key. When done, click OK and Restart. You’ll have received these details in your email upon registration.
Activating a licence #2
1. Navigate to File > Settings > Proxy.
Navigating to proxy config
2. Insert your proxy details. Make sure to tick the Use Proxy Server box.
Residential Proxies
Address: pr.oxylabs.io
Port: 7777
You can also connect to a proxy in a specific country. For example, if you enter us-pr.oxylabs.io under Address and 10000 under Port, you’ll connect to a US-based proxy server. Please refer to our documentation for a complete list of country-specific entry nodes or if you need a sticky session.
Residential Proxies setup
Enterprise Dedicated Datacenter Proxies
Specify the following if you purchased Dedicated Datacenter Proxies via sales.
Address: 1.2.3.4 (a specific IP address)
Port: 60000
You’ll have to choose an IP address from the acquired list. Visit our documentation for more details.
Self-Service Dedicated Datacenter Proxies
Specify the following if you purchased Dedicated Datacenter Proxies via the dashboard.
Address: ddc.oxylabs.io
Port: 8001
Here, the port indicates the sequential number of an IP address from the acquired list. Check our documentation for more details.
Datacenter Proxies
Address: dc.oxylabs.io
Port: 8001
In the pay-per-IP subscription model, the port represents the sequential number assigned to an IP address from the given list, meaning port 8001 will use the first IP address on your proxy list. For more details, please check our documentation.
With the pay-per-traffic subscription, the 8001 port will pick a random IP address, but the IP will remain consistent for the duration of the session. You can also specify the proxy’s geo-location within the user authentication string, such as user-USERNAME-country-US:PASSWORD, which will connect you to a United States proxy. Please refer to our documentation for more information.
ISP Proxies
Address: isp.oxylabs.io
Port: 8001
3. Click OK and Restart to complete the proxy setup.
4. Upon starting a new crawling job, you'll be asked to enter your Oxylabs proxy user’s credentials. Use the same username and password you entered to create a new proxy user in the Oxylabs dashboard.
Authenticating proxy connection
That’s it! You can now start scraping public data without worrying about blocks.
Wrapping up
Combined with Oxylabs proxies, Screaming Frog can deliver public data in real-time while bypassing anti-bot measures.
As the tutorial shows, setting up proxies on this platform is simple, so you can start working on your SEO projects right away. If you have questions about integrating Oxylabs products, don’t hesitate to drop us a message at support@oxylabs.io or reach out via live chat.
Please be aware that this is a third-party tool not owned or controlled by Oxylabs. Each third-party provider is responsible for its own software and services. Consequently, Oxylabs will have no liability or responsibility to you regarding those services. Please carefully review the third party's policies and practices and/or conduct due diligence before accessing or using third-party services.
Useful resources
Datacenter Proxies vs. Residential Proxies
A question might have arisen whilst scrolling through the infinite pools of proxy information - “what’s the difference between data center and residential proxies?” This article tackles this question.
How Does a Proxy Server Improve Security
Find out the numerous features of proxies and how they improve security.
Dedicated IP vs Shared IP
When it comes to hiding your identity online, using proxies is a great way to stay anonymous and keep your real IP address hidden. In this article, you’ll get an answer to the question, why dedicated proxies are usually better than shared ones.
Get the latest news from data gathering world
Get Screaming Frog proxies for $8/GB