Thanks to the GIL, thread-based parallel programming in (pure) Python is not exactly a thing. Thanks to memory protection, process-based parallelism involves isolated memory per process and inevitably adds IPC overhead. Processes can not share memory – or can they? Meet POSIX shared memory!
Follow me on a lightning-fast journey through some commonly forgotten internals of modern (Unix) operating systems, recently added yet still dark corners of the Python standard library and, finally, a package deserving more attention. Learn why POSIX shared memory is simply amazing for parallel Python code. Did you know that you can actually implement a parallel for loop, similar to cython
(prange
), numba
(prange
) and openmp
(#pragma omp parallel
), in almost pure Python (well, with a hint of numpy
)?