Testing Spark applications to improve quality

Developers at Adevinta Spain wanted to run unit, integration and end-to-end tests on Spark applications. Learn how they implemented this.

 

 

A useful read for big data developers, get into the detail of how the team at Adevinta Spain implements unit, integration and end-to-end tests on Spark applications as they build a Data Platform. The full article reveals a typical project layout, how to use Spark sessions, what the various test functions enable and how this improves the quality of the code and the CI pipelines.

Highlights include:

  • Layout of a Spark project
  • How to isolate different Spark tests
  • Details of the test implementation code
  • The idea behind SharedSparkSessionHelper
  • Links to real-life Adevinta examples

 

Testing Spark applications to improve quality