We all know how to deal with data if we have them on filesystem. But when it comes to work with data that come from a streaming (like Apache Kafka), there is not as rosy. There are a lot of frameworks to work with these with Java&Scala (Kafka Streams, Spark Streaming, Apache Flink, Akka Streams), but not so many for python.

There is no need to convince you that Python is the easiest language to apply ML models or another business logic. During the presentation I will show and compare three ways of dealing with Kafka-based streams of events: Spark Streaming with pyspark, Winton Kafka Streams and low-level API from Confluent. All processing will be applied on real-time data from road traffic, including weather, road conditions, and event length of vehicles. Let’s see how python and kafka can help preventing road accidents and consequently make roads safer.