Dask scales the existing PyData ecosystem of tools. Dask is a general purpose library for parallel computing that has been used to parallelize other libraries like Numpy, Pandas, Scikit-Learn, XGBoost, and many others.

However, these parallel libraries are only useful if you both have access to parallel hardware, and the devops expertise to use it. This excludes many important communities. Over the last few years a lot of developer focus in the Dask community has gone into solving this problem with deployment libraries like dask-kubernetes, dask-yarn, dask-gateway and others, and yet deployment remains a significant accessibility hurdle today. This talk starts with some of those challenges.

Then, Dask developers joined with Python web developers to make Coiled, a global scale service providing managed Dask to everyone, everywhere. This talk describes some of our design constraints, a bit of the architecture, and then goes into examples of what this enables across different parts of the PyData community.