Join for free and connect with our local tech scene
Stay on top of the latest companies and upcoming events with our weekly newsletter, and be counted among the people building the future of your local tech community.
Topic: Productionalizing Spark streaming applications
Spark Streaming has quickly established itself as one of the more popular Streaming Engines running on the Hadoop Ecosystem. Not only does it provide integration with many type of message brokers and stream sources, but it also provides the ability to leverage other major modules in Spark like Spark SQL and MLib in conjunction. This allows for businesses and developers to make use out of data in ways they couldn’t hope to do in the past.
However, while building a Spark Streaming pipeline, it’s not sufficient to only know how to express your business logic. Operationalizing these pipelines and running the application with high uptime and continuous monitoring has a lot of operational challenges. Fortunately, Spark Streaming makes all that easy as well. In this talk, I’ll be going over some of the main steps you’ll need to take to get your Spark Streaming application ready for production, specifically in conjunction with Kafka. This includes steps to gracefully shutdown your application, steps to perform upgrades, monitoring, various useful spark configurations and more.
Presenter: Robert Sanders, Engineering Manager, Big Data Practice, Clairvoyant
Robert Sanders is an Engineering Manager at Clairvoyant. In his day job, Robert wears multiple hats and goes back and forth between Architecting and Engineering large scale Data platforms. Robert has deep background in enterprise systems, initially working on fullstack implementations and then focusing on building Data Management Platforms.
Location: Theatre Room - The University of Advancing Technology