Back to all stories

Taming data workflows with a move to Athena

  • Our Brands
  • Tech

Find out how leboncoin shifted from Apache Kafka to Athena to better leverage high volumes of data.

After many years of using Apache Kafka, the tech teams at leboncoin needed to rethink workflows in order to be able to analyse, transform and consolidate events in “offline” data storage. Learn how they went from a tedious, manual process to a fully automated workflow, using AWS S3/Athena/Spark, minimising the amount of human input required.

Highlights include:

  • A deep dive into our team’s journey from the Kafka “hot” to S3/Athena “cold” storage workflow
  • How they programmed a solution to fully automate an error-prone archiving process
  • The technical and organisational challenges of ensuring consistency
  • Examples of connector configurations, base configuration and error handling
  • How monitoring and alerting keeps our datalake “fresh” for our data scientists, analysts and data services

Read the full article here.

Related stories

Discover all stories

Building a better view of trends

Read more about Building a better view of trends

Reducing the pain of UI testing

Read more about Reducing the pain of UI testing

Agile feedback at leboncoin

Read more about Agile feedback at leboncoin