Slides to the Lecture: http://bit.ly/1bmOlEX
Totango transforms raw usage data, key-value attributes and configuration into actionable analytics and visual insights.
Our batch processing chain is based on a collection of dozens of commands, whose dependencies are managed by Luigi, an open source framework developed by Spotify.
Each command’s ETL is based on Apache Spark functional computation on collections of structured JSON documents stored on Amazon S3.
This architecture is a great improvement over our legacy batch architecture, as it has very convenient and strong dependency management that is functional and dictated by data input/output flow of immutable collections.
The talk will discuss architectural decisions, challenges in implementation, lessons learned, and transition plan from a legacy architecture. All these while maintaining flexibility to allow ongoing changes and product additions.
