Amazing things your (Unix) operating system can do for you: POSIX shared memory

Sebastian M. Ernst

Prior knowledge:
Previous knowledge expected
General knowledge of Python and `numpy`. Basic experience with parallelism in Python recommended.

Summary

Thanks to the GIL, thread-based parallel programming in (pure) Python is not exactly a thing. Thanks to memory protection, process-based parallelism involves isolated memory per process and inevitably adds IPC overhead. Processes can not share memory – or can they? Meet POSIX shared memory!

Description

Follow me on a lightning-fast journey through some commonly forgotten internals of modern (Unix) operating systems, recently added yet still dark corners of the Python standard library and, finally, a package deserving more attention. Learn why POSIX shared memory is simply amazing for parallel Python code. Did you know that you can actually implement a parallel for loop, similar to cython (prange), numba (prange) and openmp (#pragma omp parallel), in almost pure Python (well, with a hint of numpy)?