Probabilistic programming is sometimes referred to as “modeling for hackers”, and has recently been picking up steam with a flurry of releases including Stan, PyMC3, Edward, Pyro, and Tensorflow Probability

As these and similar systems have improved in performance and usability, they have unfortunately also become more complex and difficult to contribute to. This is related to a more general phenomenon of the “two language problem”, in which performance-critical domain like scientific computing involve both a high-level language for users and a high-performance language for developers to implement algorithms. This establishes a kind of wall between the two groups, and has a harmful effect on performance, productivity, and pedagogy.

In probabilistic programming, this effect is even stronger, and it’s increasingly common to see three languages:

• One for writing models
• A second for data manipulation, model assessment, etc
• A third for implementation of inference algorithms

Solving this “three-language problem” usually means accepting either lower performance or a restricted class of available models and inference algorithms.

It doesn’t have to be this way. The Julia language supports Python-level coding with C-level performance. In Julia, Julia’s own code is “first-class”: code can be pulled apart and manipulated as a data structure. This leads to an approach for high-level representation of models, with transformations and optimizations specific to a given model or inference family.

This is the approach taken in Soss, a small and extensible Julia library that provides a way to represent and manipulate probabilistic models. Inference methods in Soss can be developed completely independently and without constraints, allowing arbitrary algorithms with no inherent ceiling on performance.

After a brief introduction to probabilistic programming, we will explore the approach taken by Soss, its current state, and the path forward.