An alternative approach to A/B test results calculation: Streaming

  • twitter
  • linkedin
  • facebook
  • email

A/B test results are an essential tool for delivering continuous improvement. If you don’t know how a proposed modification performs against the original state, how can you be sure it’s actually an improvement?

By Galderic Punti

The trouble is, A/B tests run in the cloud tend to rely on batch queries which are relatively expensive. So is it possible to calculate those results in a pure streaming mode by processing each event only once?

In this article we explain how the Adevinta team built a streaming mode for A/B test results:

  • How we applied Welford’s algorithm to calculate mean and standard deviation in a single pass (and the problems this caused).
  • How we refined our system to prevent potential data loss with a weighted version of Welford’s algorithm.
  • How to scale the streaming system to support a large number of simultaneous experiments.
  • How to avoid blocking caused by synchronous calls.

Ready to see how it’s done? Read the full article on the Adevinta tech blog.

Our Jobs

View all our jobs

More resources

Discover our media resources, brand assets, guidelines, photos and more