Description

Researchers, data scientists, and others; often encounter problems, which are parallel in nature. Despite this, many find it difficult to take advantage of the potential speedup afforded by parallelism, especially in the early stages of model development and data analysis. In such cases, one may write sequential code that only deals with a chunk of the data at a time, and then the parallelism is implemented once the code is moved into production. This approach is not ideal, as the original code written by the researcher will be slower, and may not map well onto the resulting production code. Thanks to technologies like IPyParallel it has become much easier to use parallelism in research, and this tutorial will demonstrate how that can be done.

Abstract

Using a Jupyter notebook, several examples of data analysis and model fitting tasks will be demonstrated. Then the next step will involve showing how they can be implemented in parallel with relatively small changes to the logic of the code.

• Part 1 (20 - 30 minutes):

  • Introduction of the Jupyter notebook environment with code snippets, and mock data
  • The mock data will be split into n chunks, each of which will require its independent analysis, the results of which will later be pooled together
  • Three main data analysis tasks will be shown in increasing order of complexity (code for these will be provided):
    • Summary statistics
    • Fitting a statistical distribution to the data
    • Fitting a simple Neural Network

• Part 2 (30 minutes)

  • Basic background on parallelism, how it can be implemented in Python, and how it may differ from concurrency
  • Introduction to IPyParallel and some background on using it including common pitfalls

• Part 3 (30 - 40 minutes):

  • Mapping what was conveyed in Part 2 to what was demonstrated in Part 1
  • The data analysis tasks that were executed in part 1 will be cast into a parallel setting using IPyParallel
  • Run-time performance benefit will be shown, as well as the effect on overall CPU utilization
  • Comments will be made on how the code was structured in Part 1 in a specific way to make it easily handled in parallel later, and how only certain types of problems can be simplified in this manner

After the tutorial, the participants should have an idea of how to parallelize their analysis, and feel confident to try it for themselves. They should see how using IPyParallel in their research can benefit both them and those who will have to put their code into production later on. They should also have an understanding of the limitations of this process and differentiate between the problems where this can and cannot be used.

The materials will all be uploaded to Github in the days prior to the tutorial so the participants can use it to follow along better.

Target audience: The audience must be very comfortable programming in Python to be able to follow the examples. They must also have basic knowledge of data analysis and model fitting, to be able to grasp the context of the tutorial.