Multi-tenant Spark workflows in Auto Scalable Mesos clusters

Tue, 06 Nov 2018 12:00:00 +0000

Recommendation algorithms have been the core of the Netflix product from very early on. Because of their importance, we continually seek to run our machine learning workflows in a reliable, scalable and robust manner.

We will present our design choices on building a Mesos-centric multi-tenant architecture for running Spark-based machine learning workflows that power the algorithms behind Netflix recommendations. Also we will share our experience using the auto-scaling capabilities of Amazon Web Services to dynamically change the size of our clusters to support the allocation of thousands of spark jobs running daily. We will discuss how we are leveraging Apache Spark to deploy batch jobs as well as the interactive use of Zeppelin Notebooks efficiently in this shared environment.

Mesos at opentable

Thu, 20 Aug 2015 12:00:00 +0000

Opentable has been using Apache Mesos for production workloads and for running critical parts of their production services for more than a year.

Not only did Mesos help deploying resilient / elastic standalone applications and services , but also the distributed / fault-tolerant frameworks like Apache Spark for Data processing and machine learning. Mesos enabled Opentable to run multiple distributed applications across the same infrastructure at scale.

Pablo will tell the story of how Opentable started with Mesos, the pain points of dealing with an hybrid Mesos + non-Mesos environment and how to survive in the transition.

Mesos on Share what you know

Multi-tenant Spark workflows in Auto Scalable Mesos clusters

Mesos at opentable