Find out how leboncoin shifted from Apache Kafka to Athena to better leverage high volumes of data.
After many years of using Apache Kafka, the tech teams at leboncoin needed to rethink workflows in order to be able to analyse, transform and consolidate events in “offline” data storage. Learn how they went from a tedious, manual process to a fully automated workflow, using AWS S3/Athena/Spark, minimising the amount of human input required.
Highlights include:
- A deep dive into our team’s journey from the Kafka “hot” to S3/Athena “cold” storage workflow
- How they programmed a solution to fully automate an error-prone archiving process
- The technical and organisational challenges of ensuring consistency
- Examples of connector configurations, base configuration and error handling
- How monitoring and alerting keeps our datalake “fresh” for our data scientists, analysts and data services
