PyData Warsaw 2017 - Presentation: Conduct a computational experiment in 10 minutes (instead of 1 week) & process results in real-time

Friday 15:35–16:05 in Track 1

Conduct a computational experiment in 10 minutes (instead of 1 week) & process results in real-time

Laurent Picard

Audience level:: Novice

Description

By distributing a Python algorithm over 1,000 computers, a 1 week computation can drop down to 10 minutes only. Moreover, storing and processing the results can be complex and time consuming, especially with large amounts. This can be seamless and real-time. Discover the power of Spark and BigQuery.

Abstract

During this talk, you'll see how to:

Adapt a Python algorithm for parallel computing
Distribute it over 1,000 cores with Spark
Stream the results in real-time with BigQuery
Analyze the results, even while the experiment is still ongoing

Subscribe to Receive PyData Updates

Tickets

Get Now