A/B test results are an essential tool for delivering continuous improvement. If you don’t know how a proposed modification performs against the original state, how can you be sure it’s actually an improvement?
The trouble is, A/B tests run in the cloud tend to rely on batch queries which are relatively expensive. So is it possible to calculate those results in a pure streaming mode by processing each event only once?
In this article we explain how the Adevinta team built a streaming mode for A/B test results:
- How we applied Welford’s algorithm to calculate mean and standard deviation in a single pass (and the problems this caused).
- How we refined our system to prevent potential data loss with a weighted version of Welford’s algorithm.
- How to scale the streaming system to support a large number of simultaneous experiments.
- How to avoid blocking caused by synchronous calls.
Ready to see how it’s done? Read the full article on the Adevinta tech blog.