In three years, our servers scaled from processing zero to 500,000 queries per second. With 2000% growth in 2011, tens of millions of dollars of transactions taking place daily, a data pipeline bursting at the seams, and 24/7/365 uptime, AppNexus engineers faced a task likened to changing the engine of a 747 in midflight.

This talk will explore how we horizontally scaled our pipeline and data reporting by adapting a variety of technologies, including Hadoop, Vertica, NoSQL, MySQL, and more. In this session you will gain from our experience and hear our lessons-learned in terms of hardware application, monitoring, rollback, failover nodes, controlling memory allocation, and day-to-day fire drills. We will cover the data flow between different systems, integration details, job scheduling, common pitfalls, and the finely tuned solutions that we have developed.

Mike Nolet

CTO & Co-Founder,
AppNexus

Mike oversees the Engineering and Technical Operations teams, having helped design the AppNexus system that handles over 27 billion ad requests a day. Prior to founding AppNexus, Mike was Director of Analytics and Product Manager at Right Media (acquired by Yahoo! for $850 million in July 2007), where he designed and managed key applications, including predictive optimization, inventory forecasting, and Media Guard, which protects publishers from harmful or unwanted ad content. Mike's deep understanding of the intersection between business needs and technology innovation began at 15, when he sold his first software application, and continued at Princeton University, where he studied Economics. On his popular blog, MikeOnAds, Mike analyzes the fast-changing display ad management and buying space, and the dynamic interplay between ad networks, ad exchanges, real-time bidding and the entire ecosystem.