"When I joined in 2013, Foodpanda, one of the many Delivery Hero brands, was doing a few hundred orders a day," says Mathias Nitzsche, VP of engineering at Delivery Hero. "Today we’re doing hundred thousands of orders a day, still growing more than 100% a year."
With that kind of growth, the ability to scale is everything. To do that, Mathias and his team need to clearly see into what was happening across their infrastructure and applications. "For us, speed is not as important as scale, and scale is nothing without visibility," he explains. "It is impossible to run a platform like ours without visibility."
Today, Delivery Hero’s engineering organization has more than 1,000 developers working across 15 platforms. Pandora, the internal name for their largest platform, powers Foodpanda, Foodora, and a few other brands.
Meaningful visibility to inform business excellence
To get valuable business insights, Delivery Hero relies on New Relic. Originally adopting New Relic to monitor application availability and performance, the company has since expanded its usage to monitor business-critical key performance indicators (KPIs)—from the number of restaurants requested per area, to orders per platform and country, to payment charge-back rates per payment provider.
Dashboards are the most critical component, says Mathias. All Delivery Hero teams rely on dashboards daily to understand performance. With over 500 applications to oversee, Mathias says, "nobody has the time to look through an individual application in any detail." Cross-application dashboards give the team a high-level view of applications and help track business metrics like the number of orders processed globally and the number of errors found in their applications.
"Without dashboards, our monitoring would be only half as useful. Dashboards is what makes New Relic beautiful," he says. And because this is connected to every product in the New Relic platform, engineers can stream and track data with APM, for deeper analysis, segmentation, and filtering within dashboards."
New Relic provides both the narrow focus and broad overviews Mathias’ team needs. "We have more than one hundred dashboards created by engineers, QAs, and product managers, shown on many screens all over the office," he says. These dashboards provide granular insight into a range of business indicators, displaying the information that matters most at any given time.
Many tools provide a lot of technical telemetry, but New Relic can connect that with business metrics and costs.
From monolith to microservices to DevOps
Pandora's IT infrastructure has also undergone a tremendous transformation. To better support business growth, the Pandora team migrated their infrastructure to a microservices architecture, running on Amazon Web Services (AWS).
Mathias' team used New Relic to monitor its microservices migration in real time. The team continues to use New Relic to monitor other migrations from new acquisitions, or migrations from various regional platforms to its global platform. For example, the company migrated its Finnish and Swedish applications to the global platform so the teams serving those countries could better leverage core Delivery Hero services like search, payments, and infrastructure. During these rollouts, teams use New Relic to monitor things like speed, number of requests and errors, and database queries. "You double the traffic in some of these rollouts, and you want to see how it behaves," Mathias says.
Mathias' team now manages hundreds of repositories spread across dozens of microservices running in their Kubernetes clusters.
Today, Delivery Hero is a true DevOps company, split into cross-functional teams. Rather than being structured by IT functions, they’re structured according to the services they manage. For instance, there is a payment team, a checkout team, and a search and discovery team, among others. Each team oversees its own product design, its own frontend and backend development, and its own infrastructure resourcing.
This transformation has helped Delivery Hero scale the engineering teams, and as a result their DevOps culture has flourished, says Mathias. Now all teams are cross-functional and focused on the direct needs of the business. "We conduct a lot of sessions to show what other teams are doing in terms of monitoring and looking past errors," says Mathias. "We can then use that information to accelerate development processes and decision-making."
Eye-opening revelations
Dashboards has helped Delivery Hero connect technical metrics to cost optimisations. "Many tools provide a lot of technical telemetry," Mathias says, "but New Relic can connect that with business metrics and costs."
Mathias says another huge benefit of New Relic is that it allows him to view the team’s infrastructure consumption and then use that data to optimise their environment. Mathias' biggest revelation came when he realised he could use New Relic to optimise the size of the Pandora Kubernetes cluster. Before using New Relic, it wasn't easy to determine which applications consumed which cluster resources. For example, in July 2018, Pandora’s biggest app used 700,000 distributed compute units; but after monitoring it with New Relic, they optimised it to use only 200,000 units—that’s a 71% reduction in costly resources!
"Without New Relic, we wouldn’t have known where to start’, Mathias says. ‘We now use less than half of the compute units we used a year ago. It was an eye-opener in terms of how much visibility we could get into our AWS consumption."
