After briefly introducing Yelp and more specifically click-through rate (CTR) prediction at Yelp, we will start out with a basic setup for model-based predictions in a production system. From there we will point out deficiencies of said setup in various areas, some of which arise especially in large scale environments or when predicting CTRs.

This will give us an opportunity to dive deeper into a selection of insightful practical lessons we've learned from operating and scaling the mission-critical CTR prediction system at Yelp. Those lessons can be categorized into:

• Infrastructure
  • For example: How can you set yourself up for a successful integration of user feedback?
• Model Comprehension
  • For example: Do you treat your model as a black box or do you know the importance of each feature? Why might that be valuable?
• Specifics
  • For example: When applying a threshold, wouldn't missing training data below the threshold affect the online model performance in that area? (Spoiler: Yes, it does.)

Along the way, various Python tools which we're actively using will be highlighted.

By touching on a multitude of diverse challenges, this presentation strives to provide valuable insights for any engineer interested in or working with ML models in a production environment.