Glen De Cauwsemaecker
Lead Crawler Engineer at OTA Insight
Glen De Cauwsemaecker is the Lead Crawler Engineer at OTA Insight (around 350 employees, of which 67 engineers). With over ten years of experience, he drives the technical roadmap and innovation with a team of four other engineers. Besides researching, developing, and maintaining crawlers, he also maintains and creates new projects related to proxy/networking services, backend services, libraries, automated browser solutions, fingerprinting, tooling, and a trivia of cross-platform and language agnostic reverse engineering activities.
Learn more about Glen's presentation:
How to Continuously Yield High Quality Data While Growing From 100 to 100M Daily Requests
The company where Glen works, OTA Insight, gathers public data from hundreds of different kinds of data sources, with over 100M requests daily, amounting to approximately 9000 GiB (compressed) of raw data. This presentation will cover the story of how OTA Insight grew from 100 daily requests to 100,000,000 requests while continuously yielding high-quality data every single day.
Glen will cover a variety of topics, including:
The beginning of the OTA Insight company and how they scaled up their business, as well as what they're working on today to prepare for tomorrow;
Company's infrastructure and how it's involved to help them with the growth;
What technology do they use, how did it evolve, as well as what choices did they make at every step;
Where they see themselves as a crawler team, what challenges they foresee for the future, and how they aim to stay on top of the game.
Meet OxyCon 2022 speakers
Coming from different industries and backgrounds but united in common passion and aspirations, these web scraping experts will share their experiences and answer your questions.
GET IN TOUCH
General:
hello@oxylabs.ioSupport:
support@oxylabs.ioCareer:
career@oxylabs.io
Certified data centers and upstream providers
Connect with us
Advanced proxy solutions
Resources
Innovation hub