Sometimes the business and technical requirements make the design of a new system nearly impossible or excessively complex. Python's elegance and simplicity helped to build a data pipeline that processes Terabytes per day while becoming the 'lingua franca' for every technical person in the company.
Sometimes building a new system can be an even more daunting task than it sounds if you have to comply to the regulations of a highly regulated industry like Telecommunications, which also comes with huge volumes of data in high velocity.
We needed to move beyond the Proof-Of-Concept where the company's analytics suite was before and move to a architecture which could be scaled, be highly configurable to comply with all the legal and privacy requirements in every country, allow the data scientists to perform their analysis in the data, enable the development of novel data products and do it with a team that lacked experience in big data solutions.
Why Python end-to-end?
Python proved to be the solution that through its simplicity allowed to build a system that matched the requirements, ease its deployment and be the 'linqua franca' for the company's engineers.
We believe that Python can help many companies to build successful products and many engineers and data scientists to enjoy doing it and we want to share our experience on doing it.