Scaling a data pipeline is easy, but scaling a data pipeline team? Hard stuff. Money and hardware don’t fix these problems. As your team expands and codebase grows you split your big team team into micro-teams but subtle issues arise. We’ll cover the backbone of data pipelines (Kafka, SQS), enumerate the problems newly split teams run into, examine how to solve those problems using innovative solutions, like SNS message filters, and common solutions, like extra Kafka topics. I’ll share success stories, like working communication, and stories-to-learn from. Attendees will leave understanding why and how to split data pipeline teams, the types of problems they might encounter, and confidence to pull it off.
From Big Data to finance to the console video game industry, Brad’s seen diverse software systems. He’s contributed to 25+ OSS projects, and, as a Principal Engineer at Sonatype, leads a data team specializing in OSS identification. He is a veteran speaker, clocking in over 50 appearances from small audiences to those numbering in the hundreds.