We are witnessing an unprecedented progress in the area of Machine Learning (ML), in large part powered by the Deep Learning methods that in recent years have taken center stage from the classical ML approaches. These powerful algorithms have been successfully applied to a number of various applications, and have proven very useful in the context of autonomous driving as well. This holds especially true when it comes to the problems of perception and prediction, two critical parts of the self-driving puzzle. In this talk I will present some of the deep learning efforts at ATG revolving around these two tasks, and how they can be used to improve safety and performance of the self-driving technology.