In this talk I will discuss the way data-driven products are built Allegro with examples from image quality classification up to a search relevancy pipeline. Topics covered are: Test or data driven development (a.k.a Jupyter or IDE) Airflow as a Data workflow and scheduler platform Example project pipelines Lessons learned
In this talk I will discuss the way data-driven products are built Allegro with examples from image quality classification up to a search relevancy pipeline. Topics that will be covered are: Test and data driven development: How to get/go from a jupyter notebook to a production scheduled batch job Automated testing of pySpark jobs Spark jobs - Python or Scala? Airflow as a Data workflow and scheduler platform Metrics and monitoring Example project pipelines * Lessons learned