Real-Time Crawler Sees a Further Enhancement

Vytautas Kirjazovas

2020-01-162 min read

Across 2019, dozens of developments have been added to our ever-evolving Real-Time Crawler. All of these additions have been designed to improve existing functionality and help our partners to increase the efficiency and productivity of their daily web scraping operations.

With less than a month in 2020, we are proud to announce that Real-Time Crawler sees a further enhancement, which will bring additional value to our partners.

From now on, our partners can pass real-time request parameters via URL, and retrieve the data instantly on the same connection, either in raw HTML or parsed in JSON. This enhancement will result in further ease of use of our solution and save additional time when setting up web scraping assignments to Real-Time Crawler.

URL example

https://realtime.oxylabs.io/v1/queries?source=search_engine&query=adidas&parse=true&access_token=12345abcde

{
   "results":[
      {
         "content":{
            "url":"https://www.search-engine.com/search?q=adidas&hl=en&gl=US",
            "page":1,
            "results":{
               "paid":[
                  {
                     "pos":1,
                     "url":"https://www.stylight.com/Adidas-Originals/",
                     "desc":"Top Offers on the Latest Trends. Great Selection & Free Shipping. Sale & Discounted Items. Latest Looks for 2018. Selection for Men & Women. All Styles & Occasions. New Products Added Daily. Styles: Trendy, Designer, Boho, Casual, Elegant, Hipster.",
                     "title":"Adidas Online Shop - stylight.com\u200e",
                     "url_shown":"www.adidas.stylight.com/",
                     "pos_overall":1
                  }
               ],
               "images":[
                  {
                     "alt":"Image result for adidas",
                     "href":"/search?q=adidas&hl=en&gl=US&tbm=isch&source=iu&ictx=1&fir=TXOvv4Rs6l-N_M%253A%252COHhi19nLaZbr8M%252C_&vet=1&usg=AI4_-kRzdn_gBZA3lX8bH-1HC-KlE2IF2g&sa=X&ved=2ahUKEwjE_sWwp4rnAhVRRqwKHSdrDdwQ9QEwHnoECAQQBQ#imgrc=TXOvv4Rs6l-N_M:",
                     "source":"https://www.adidas.com/us/superstar-shoes/FV3284.html"
                  },
                  {
                     "alt":"Image result for adidas",
                     "href":"/search?q=adidas&hl=en&gl=US&tbm=isch&source=iu&ictx=1&fir=k4rfhqSTMHnZ5M%253A%252C_pzST5cq041pdM%252C_&vet=1&usg=AI4_-kSRvXEyK9M7WzIvNaW18mbrQ9yT9A&sa=X&ved=2ahUKEwjE_sWwp4rnAhVRRqwKHSdrDdwQ9QEwH3oECAQQBw#imgrc=k4rfhqSTMHnZ5M:",
                     "source":"https://www.adidas.com/us/stan-smith-shoes/M20324.html"
                  },
                  {
                     "alt":"Image result for adidas",
                     "href":"/search?q=adidas&hl=en&gl=US&tbm=isch&source=iu&ictx=1&fir=PSMHk-ACViSNlM%253A%252CjiWLx5Hit80izM%252C_&vet=1&usg=AI4_-kSid_bnps2Tx3SH4AZFNI9D5svARw&sa=X&ved=2ahUKEwjE_sWwp4rnAhVRRqwKHSdrDdwQ9QEwIHoECAQQCQ#imgrc=PSMHk-ACViSNlM:",
                     "source":"https://en.wikipedia.org/wiki/Adidas"
                  },

In simplest terms, our partners will be able to specify their exact requirements without writing a single line of code. The one-line integration is based on the real-time endpoint, meaning the data will be fetched back instantly within seconds on the same request.

Please note: Data retrieval time will vary depending on the data source that is being used. For further information about the new feature, please refer to our official documentation or get in touch with your personal account manager.

What is Real-Time Crawler, and why consider It?

The web is an insightful but complex source of knowledge, as public data sources frequently change their structure. Some data-wealthy sites require advanced technical knowledge to gain access, while others add additional steps that mess up any data-gathering operation. That is why the majority of our clients turn to our Real-Time Crawler solution to gather public data from the web at scale and do so in real-time.

In short, Oxylabs Real-Time Crawler is a robust data gathering tool optimized for effortless data extraction from search engines and e-commerce sites, also referred to as real-time web scraping solution. In fact, if it caught your attention, check out our in-depth blog post on Real-Time Crawler and its advantages.

Or, if you would like to get familiarized with our solution in video format, hit the play button below to hear what our Product Owner Aleksandras has to say:

Final thoughts

Real-Time Crawler will no doubt see further additions in 2020, but already our partners are benefitting from being able to retrieve accurate web data in a hassle-free manner on demand.

About the author

Vytautas Kirjazovas

Head of PR

Vytautas Kirjazovas is Head of PR at Oxylabs, and he places a strong personal interest in technology due to its magnifying potential to make everyday business processes easier and more efficient. Vytautas is fascinated by new digital tools and approaches, in particular, for web data harvesting purposes, so feel free to drop him a message if you have any questions on this topic. He appreciates a tasty meal, enjoys traveling and writing about himself in the third person.

Learn more about Vytautas Kirjazovas

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.