Apache Spark Engineer - Platform - Senior
San Francisco, CA
The Spark Platform team is building low-latency, distributed analytics infrastructure for Bloomberg based on Apache Spark.
Instead of building dozens of isolated Spark applications for individual problem domains, we are building a platform that makes it easy for teams to plug in their business logic without duplicating common functionality. This includes connecting to various datastores or real-time streams, figuring out a way to serverize Spark transforms or having to re-implement transforms such as currency conversion that are very common in financial analytics. But we can' t just use Apache Spark as is. We need to enhance open source Spark to fit our low-latency, high throughput and high availability contexts. That' s where you come in.
As part of our team, you' ll collaborate with a number of the core Spark contributors to co-design and co-develop enhancements to Spark in areas ranging from Performance to HA.
We' ll trust you to:
- Contribute to the development of the Bloomberg Spark Platform
- Interact with application teams to understand their specific requirements and drive the next set of features for the Spark Platform
- Meet with key players in the open source community to motivate necessary development efforts
- Contribute code to the Apache Spark community in areas ranging from Machine Learning to Spark HA
- Strong programming proficiency in Java and Scala (a minimum of two years)
- An expert-level understanding of Apache Spark - your work will involve modifying its internals
- Experience with MLlib
- Open source contributions of any sort
- Professional experience developing analytics infrastructure
If this sounds like you, submit an application and watch our talk at Spark Summit East to learn more about Bloomberg' s Spark Platform (https://spark-summit.org/east-2016/events/spark-at-bloomberg/).