Armen Hamstra on How He Broke LinkedIn – and Got Promoted

Head over to engineering.linkedin.com for news, projects, and career opportunities within LinkedIn Engineering. For more interviews, click here.

It's 8:30PM on a Friday evening, and disaster recovery plans for LinkedIn’s production data centers are deployed. Major services such as the Recruiter tool, payments, and invitations have stopped working. Over 250 engineers are on deck — convening in conference rooms, connecting remotely from home. Half of them are actively monitoring and fixing the issues, while the other half are gaping in frenzy as vital parts of the site are knocked offline.

A natural disaster? Phishing? A hack? All plausible, but this downtime of major LinkedIn services was in reality the result of a concerted production failover led by then-staff (senior staff, since publication) software engineer Armen Hamstra.

In 2013, LinkedIn built the company’s first wholesale production center in Virginia to accompany our existing data center in Los Angeles. Since LinkedIn had been operating over a decade with just one data center, the company had to reconfigure its stack so it could actively serve out of multiple data centers.

While most of the code made it to being active-active, certain services remained single-master. To ensure that these single-master services would remain online even with a data center failure, the engineering team — with Armen as the lead — decided towards the end of 2014 to purposefully fail out of our Los Angeles data center into our facility in Virginia.

I sat down with Armen to discuss the single-master production failover — the logistics, the resources, and the results. We also touched on how he, as an applications development engineer, was chosen to lead a project that falls under the infrastructure scope.

Armen has come a long way from his “disastrous interview” with LinkedIn four and a half years ago to his recent promotion to senior staff software engineer and team architect — catalyzed in large part by the success of the production failover. By acting like an owner and taking intelligent risks — two of LinkedIn’s core values — we discussed how LinkedIn has helped him realize his potential and foster his engineering skills to where they are now.