How hard would it be to transform a simple pandas method to its multi-core counterpart? In this talk, we are going to look into the process of creating a library which takes a single core pandas method and transforms the method to use multi-core processing. We will dive into the world of pythons copy on write memory and pandas internals to build this small and simple extension.
Most people who worked with pandas used the pandas apply method and might have encountered it to be slow. Vectorizing your function will improve the speed issues, however, this can sometimes be troublesome and tedious. As an alternative, one can use a multiprocessing counterpart to pandas, for example, dask or modin. While these are perfectly fine alternatives, they might be overkill for executing a simple parallel apply function.
In this talk, we explore how hard it would be to create a multi-core apply method for pandas which results in increased execution speed but keeps memory overhead to a minimum. We go into the technical side of python memory management and explore copy on write memory whilst also exploring the way pandas data frames are built. In the end, this effort is combined into a simple pandas extension.
Some python code examples of how to invoke copy on write memory in python multiprocessing will be shown, as well as how the multi-core apply extension for pandas was build. Overall, this will be an interesting introduction to python multiprocessing and learning some of the internals in pandas.