For the past few months, we have been working on some changes for our Real-Time Crawler product. We are happy to finally announce that starting today, our scraper will be switching its focus (and its name) and will be split into three distinct Scraper APIs.
Our product was great and it served its purpose well, but it was time for a change. In today’s blog post, we’ll explain why we’ve decided to evolve it.
So what has changed?
Real-Time Crawler was one of the first data gathering tools in the web scraping industry that focused on gathering e-commerce and search engine data on a large scale. It helped a number of companies to easily acquire public data, but now we are making it even better both internally and externally.
We distinguished Real-Time Crawler’s capabilities and developed it into three distinct scrapers covering specific targets. This allows us to focus on more agile product development and, in turn, improve the overall performance and user experience for our clients.
So, from now on, Real-Time Crawler becomes a group of powerful Scraper APIs, each using its own edge for the best results on the corresponding targets:
Refined product features
Scraper APIs were developed with more expressed features and benefits in mind. All Scraper APIs guarantee:
- 100% data delivery
- Patented proxy rotator
- High scalability
- Easy integration
- 102M+ proxy infrastructure
- Data delivery in your preferred way (AWS S3 or GCS)
- 24/7 support.
Each product shines through with specific benefits:
SERP Scraper API:
- Localized search results
- Live and reliable data
- Resilient to SERP layout changes
E-Commerce Scraper API:
- 1000s of e-commerce websites available for scraping
- Adaptive Parser
- Structured data in JSON
Web Scraper API:
- Customizable request parameters
- JavaScript rendering
- Convenient data delivery
Best part is that with Scraper APIs, you can forget about proxy management and focus on data analysis.
New look
All Three Scraper APIs can now be identified by their updated logos matching the targets they cover:
Same integration methods
To keep it simple for our clients and all those who tried out our products before this update, we decided to keep the same integration and authentication methods, as well as the same request parameters.
SERP Scraper API authentication method
SERP Scraper API uses basic HTTP authentication that requires a username and a password. It’s one of the simplest ways to start using this tool. The code example below shows how to send a GET request to scrape data from a search engine using the Realtime method.* For more information, read SERP Scraper API quick start guide.
curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "SEARCH_ENGINE_search", "domain": "com", "query": "shoes"}'*For this example, you need to specify the exact source to find available sources. Please refer to the SERP Scraper API documentation.
E-Commerce Scraper API authentication method
E-Commerce Scraper API employs basic HTTP authentication which requires a username and a password. This is the easiest way to get started with the tool. The code example below shows how you can send a GET request to books.toscrape.com using the Realtime delivery method. For more information, read E-Commerce Scraper API quick start guide or E-Commerce Scraper API documentation:
curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "universal_ecommerce", "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "geo_location": "United States", "parse": true, "parser_type": "ecommerce_product"}'Web Scraper API authentication method
Web Scraper API employs basic HTTP authentication which requires a username and a password. This is the easiest way to get started with the tool. The code example below shows how you can send a GET request to https://ip.oxylabs.io using the Realtime delivery method. For more information, read Web Scraper API quick start guide or Web Scraper API documentation:
curl --user "USERNAME:PASSWORD"'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "universal", "url": "https://ip.oxylabs.io"}'Wrapping up
Some changes all over the web and communication will happen over some period of time. All of these changes should be completed by the end of this year.
If you have any questions or suggestions regarding Scraper APIs changes, please feel free to contact us either via your Dedicated Account Manager or hello@oxylabs.io.
